Modern GPUs, whether they’re just sitting there idling or running hot during inference, cost enormous sums to keep them from melting into a puddle of lost throughput potential.
Those GPUs — drawing more than a kilowatt per card, with racks running into the hundreds of kilowatts — require specialized data center architecture with liquid cooling systems. They also run the risk of sitting largely idle when running low-latency applications that demand constant trips to HBM without using all the available firepower.
Fortunately, modern memory-optimized accelerators carry a significant advantage: they’re PCIe-based and cooled by air instead of liquid. They can run in legacy data centers and new ones alongside GPUs, without the need to build out liquid cooling systems and throw more GPUs (and more heat) at the problem.
Those pipelines put the memory-bound work on hardware that runs cool, let the GPUs you already have focus on the compute-heavy work they’re built for, and scale inside the power and thermal envelopes that already exist. The addressable world for liquid-cooled inference is a few hundred facilities.
Memory-optimized accelerators can both handle simple steps in agentic pipelines and accelerate frontier models with techniques like speculative decoding and attention-FFN disaggregation. That means making better use of existing GPUs and scaling up overall throughput without the constant need for liquid cooling.
Instead of installing new liquid-cooled racks, you can just fill open PCIe slots to build a disaggregated AI inference pipeline. The addressable world for air-cooled inference is every server room on Earth.
Where the weights live is where things run hot
AI inference decoding relies on rapid trips to and from wherever the model weights and KV cache of that inference live. In classic GPU architecture, both live on HBM — and every new token in a decode requires reading them all over again. Each of those round trips consumes energy, and that energy leaves the system as heat.
Keeping latency low means making those round trips as fast as possible. With GPUs, the only real lever is more aggregate HBM bandwidth — which means more GPUs, more power wasted, and more heat to manage.
Memory-optimized accelerators move those weights and KV cache closer and shorten the physical distance by keeping the weights and KV cache on-chip in SRAM. Less distance means less energy per access, which means less overall heat — and less cooling required.
The round trips are shorter, faster, and better saturate compute. Both types of cooling are bound by the laws of physics—SRAM moves less stuff around shorter distances, generating less heat. Liquid cooling is left fighting a much harder battle against the laws of physics.
How air cooled AI accelerators change the perf/TCO calculus
Running smaller models in agentic pipelines and scaling those applications isn’t the only option. Those same air-cooled systems can accelerate inference pipelines without the need to add new cooling capacity.
Disaggregated pipelines that make GPUs more efficient also address that cooling problem by effectively doing more while consuming the same amount of power. Rather than adding more GPUs and the cooling and power requirements for them, you’re adding air-cooled SRAM-based accelerators at a fraction of the wattage.
There are two current routes to making those GPUs more efficient:
- Speculative decoding runs a smaller and more efficient model alongside the GPU to “guess” the next token with lower accuracy. That model lives and runs in SRAM, while the GPU is checking the work in large batches rather than sequential decoding.
- Attention-FFN disaggregation moves the feed-forward network—the part that isn’t dealing with an ever-growing KV cache—onto an SRAM-based accelerator. The fixed size of a FFN requires scaling out a single SRAM pool, leaving the attention process to GPUs.
Both represent the same kind of opportunity: using advanced decoding techniques to do more with what’s already plugged in.
The new data center math with disaggregated AI pipelines
Cooling systems are a facilities-level design decision in new (and upgrading) data centers. Rolling out those new GPUs comes with the extra tax of hotter buildings and more complex (and expensive) cooling systems.
The buildout is also on the scale of years, not months, and additive liquid cooling adds additional points of failure to the data center. And the cardinal sin of an AI-powered applications (like most applications) is being unavailable at the times you need it most.
The vast majority of data centers weren’t built with that in mind. Instead, that first explosion of data center buildout was optimized for the previous generation of cloud computing with PCIe-based lower-density racks.
That means all the excess available capacity in those data centers are simply more slots for air cooled SRAM-based accelerators without any extra lift. The opportunity to escape the inference crunch while building snappy user experiences is the millions of racks that already have a fan.
