Frontier-level models are now blowing past the trillion-parameter mark. And despite significant optimizations in inference, the urgent need for memory capacity is exploding with it.
While small models continue to improve, high-bandwidth memory based on DRAM on classic GPUs is still the go-to for running these colossal models. Advanced disaggregated pipelines with SRAM-based accelerators can massively accelerate those trillion-parameter models. But these models are inevitably memory-bound—and require rethinking the architecture again.
DRAM offers the best possible route to getting up to the capacity required for a model well over 1 trillion parameters. But maintaining that ultra-premium user experience demands a new methodology around DRAM.
One answer—which sounds alarmingly simple aloud but devastatingly complex on silicon—is literally putting another layer on top of it. Taking a 3D approach to DRAM gives the best of both worlds: a much larger memory pool without the enormous tradeoffs of adding new GPUs.
How we got to the trillion-parameter club
OpenAI’s GPT-4 model was roughly estimated to be at around 1.7 trillion parameters with a mixture of experts architecture. At the time this was massive and considered enormously inefficient, leading to rate limits, downtime, and unreliability. In short: incredible model, enormously impractical.
The universe of high-performance frontier level models shrank significantly in response to that obvious inefficiency. The hardware simply wasn’t there yet—either in terms of raw performance or the quality of the experience it could deliver. Top models like the early DeepSeek and Qwen series models came in at the low hundreds of billions of parameters, and smaller, highly capable models emerged in the sub-70B parameter range.
The mixture of experts architecture became the norm in those low-hundred-billion parameter models. It enabled those models to run hyper-efficiently by limiting the number of active parameters, but immediately ran into the next problem: the memory wall. The compute speeds were there and HBM could hold the models—it just couldn’t feed them fast enough.
Fast forward to today, and we’re back past a trillion parameters. The latest Kimi series model, K3, sits at 2.8 trillion parameters, while DeepSeek’s latest model is 1.6 trillion parameters. The compute power and inference techniques are available to make these models performant—but further exacerbate the tradeoffs of throughput and latency in standard HBM-based architectures.
Fighting the tradeoff with a second story
While these models are colossal, they only have a smaller subset of parameters active at a given time — K3, for example, activates 104 billion parameters per token. The compute firepower then is less of a limiting factor than the clogged-up memory lanes. GPUs have historically had to reconcile the tradeoff between the two: more utilization sacrifices latency, while faster experiences leave GPUs underutilized.
Disaggregated pipelines help alleviate this by bringing the memory pool directly adjacent to the compute cores for parts of the inference process through techniques like attention-FFN disaggregation and speculative decoding. That enables those models to tap the speed that SRAM offers, enabling a premium user experience without the need to add significantly more GPUs.
But maintaining that premium experience as models explode quickly runs into one of SRAM’s disadvantages: it’s extremely fast, but it isn’t huge. The SRAM can scale out, but it can’t elegantly reach the massive pool sizes offered by HBM-oriented architecture.
Revisiting what makes DRAM great in AI inference, it turns out, isn’t a bad idea—as long as we aren’t adding more cards with just a single layer of DRAM.
Rather than rely solely on disaggregated pipelines with SRAM-based accelerators, these pipelines can integrate a new class that enables more than 10X the memory on a single card by stacking DRAM vertically. Fewer cards per inference means fewer card-to-card hops—and every skipped hop is bandwidth and energy that remains in the pipeline.
Stacking DRAM vertically takes the best of both worlds: there’s less physical distance to travel in the first place while still having access to a large pool of memory to hold the full expert library and warm state.
Chasing the future by learning from history
History doesn’t always repeat itself, but it certainly does rhyme. The industry’s first trillion-parameter model arrived well before the infrastructure was there to support it and scale it, so models shrank to align with what was available.
But the models are growing again, and this time hardware is the one enabling that growth. Larger memory pools can support a much larger model, and implementing disaggregated pipelines ensures that they are taking the best advantage of the compute that’s available—both in terms of preserving latency and making enough memory available in the first place.
Those sub-100B models are still improving dramatically, and they slot perfectly into disaggregated pipelines: high-performance, small-footprint models that can handle simpler individual tasks or serve as draft models in speculative decoding. Stacking the capacity is how the architecture keeps up with those monolithic models that refuse to stop growing.
Bringing compute closer to memory is one way to break through memory-bound problems by accelerating those pipelines and making them more efficient; literally putting memory in a stack is the next way to keep up.
