Enterprises constantly have to weigh the tradeoffs between maximizing the utilization of their GPUs and providing snappy, low-latency experiences.
That tradeoff becomes an even worse nightmare when an AI application takes off and has to scale up while preserving the same user experience. But there’s a new way to maximize the value of GPUs without sacrificing functionality — and letting those GPUs focus on what they do best.
Efficient AI inference hardware, like SRAM-based accelerators, doesn’t just deliver a low-latency experience — it can help maximize the value of those GPUs through speculative decoding. Rather than a GPU producing every single result, those GPUs spend their time verifying high-quality guesses in large batch sizes.
Those same GPUs spend their energy checking the work of memory-optimized accelerators — operating at larger batch sizes and getting more value out of what’s already on the shelf.
How batching became the (not so great) tradeoff
GPUs excel at handling concurrent tasks, with the most important one being prefill. The more tasks you can set running in parallel, the more efficiently your GPU is running.
When it comes to decode, each result has to come out in sequence — dependent on the last result — rather than processing all at once. The “answer” is pretty straightforward: run as many decode sequences in parallel as possible.
Each step in that decode process, however, requires accessing high-bandwidth memory for an ever-growing KV cache. That memory bandwidth is limited — the more data you’re trying to pull out of (or shove into) the KV cache, the more time and, more importantly, energy required.
The result is that larger batch sizes do a better job of making sure a GPU is doing the maximum amount of work simultaneously. But each additional decode added to the batch slows down the decode for the entire pipeline.
Optimizing the value of GPUs depends on capturing the highest quality user experience — both latency and uptime — while maximizing the batch size to ensure GPUs are never sitting idle. That tug of war has been the defining economic challenge of AI inference — until now.
How speculative decoding saves money, not just latency
Speculative decoding involves drafting a large number of tokens that are probably the answer. That’s handled by speculator models, which are typically much smaller and much faster. And, of course, they are often wrong.
GPUs in speculative decoding aren’t handling the decoding of every single pipeline. Instead, they check the work of those speculator models. And they’re able to check that work in large batches — verifying whole runs of draft tokens in a single parallel pass — rather than grinding out one high-value token at a time.
That constant tug of war becomes a significantly different proposition as a result. Instead of optimizing batch size against user experience, a pipeline is optimizing against the accuracy of speculators — at very low latencies, in parallel. The GPU’s compute engines stay saturated doing the kind of work they were built for, and the sequential grind that kept them waiting on memory moves somewhere better suited to handle it.
The new perf/TCO math
Smaller models on memory-optimized accelerators don’t just manage simple tasks in an agentic pipeline anymore. Speculative decoding lets them drastically improve the latencies of frontier-level models running on GPUs. The accelerators take over the sequential drafting work, while GPUs keep the part that genuinely demands their firepower: verification with the most powerful model in the pipeline.
That means all those racks of GPUs, constantly drawing kilowatts of power, aren’t just sitting there waiting for commands. Those same racks are generating more throughput for the same power — serving more users, from the same fleet, inside the same power envelope.
Speculative decoding takes what’s already available and maximizes its performance. And memory-optimized accelerators — especially in PCIe form factors — slot into the open slots enterprises already have.
Disaggregation is the future of AI inference
Demand for AI inference is only going to continue rising, and there still isn’t enough compute to go around.
But disaggregated pipelines with heterogeneous hardware offer a powerful way to survive the inference crunch. Rather than purchasing an army of additional GPUs — or sacrificing some level of quality — disaggregated pipelines simply do a better job with what’s available.
That’s critical for both user experiences and scaling AI operations. You can’t procure your way out of the inference crunch. But you can architect your way out — and the GPUs already on the racks are the place to start.
