Compute has raced ahead for two decades. Memory bandwidth hasn’t. Peak server compute has scaled roughly 3x every two years, while DRAM bandwidth has managed about 1.6x over the same stretch—and that gap is now the defining constraint on AI infrastructure.
It’s called the memory wall. Processors finish their math and then sit waiting for the next batch of data to arrive, and adding more compute doesn’t shorten the wait. For frontier AI labs and hyperscalers, that idle time shows up everywhere: throughput, latency, cost per token, and power.
AI inference makes the wall worse than any workload before it. Here’s what the memory wall is, why it hits inference harder than training, and the architectural approaches emerging to break through it.
Key takeaways
- The memory wall is the growing gap between how fast processors compute and how fast memory can feed them. Compute sits idle waiting on data.
- Decode, the token-by-token phase of inference, is memory-bandwidth-bound. Training and prefill are compute-bound.
- The KV cache compounds the problem. It shares memory with the model weights, so longer context windows force a tradeoff against concurrency.
- Four architectural responses are in play—HBM scaling, on-chip SRAM, Digital In-Memory Compute (DIMC), and 3D stacked memory—each with distinct tradeoffs.
- Pairing GPUs with memory-optimized accelerators is the most practical near-term path, with each phase of inference running on the hardware suited to it.
The memory wall shows up first in your cost per token. d-Matrix Corsair accelerators run alongside your existing GPUs and take on the memory-bound work where that cost is set. Contact us
What is the memory wall?
The memory wall is the performance bottleneck that forms when a processor can compute faster than data can be moved to and from memory. The math finishes quickly, and then the chip waits.
William Wulf and Sally McKee named it in 1995 in Hitting the Memory Wall, predicting that memory access—not processor speed—would become the dominant limit on system performance. Thirty years later, AI turned that prediction into the central problem of data center infrastructure.
The core metric is the ratio of peak compute to available memory bandwidth. The higher it gets, the more of your silicon sits idle waiting for data instead of doing useful work. And it keeps climbing: Gholami and co-authors found that peak server compute has scaled 3.0x every two years over the past two decades, while DRAM bandwidth grew just 1.6x and interconnect bandwidth 1.4x. Every new chip generation adds compute faster than it adds the bandwidth to feed it.
Why AI makes the memory wall worse
AI inference widens the gap faster than any workload before it.
Frontier models now run well past a trillion parameters—Kimi K3 sits at 2.8 trillion. Most are mixture-of-experts (MoE) models that activate only a fraction of those parameters per token (roughly 100 billion for K3), which keeps compute requirements manageable. But the full model still has to live in memory, and at production batch sizes different tokens route to different experts, so a large share of the weights gets read on every decode step anyway.
Context windows have grown from a few thousand tokens to over a million, and token volumes have exploded alongside them. OpenRouter alone carried more than 100 trillion tokens in the 13 months to November 2025.
Memory supply hasn’t kept up. Industry analysis reported by Tom’s Hardware, citing the Wall Street Journal, projects that data centers will consume up to 70% of all memory produced worldwide in 2026, and DRAM contract prices rose roughly 60% quarter over quarter in Q2 2026. The wall isn’t only about bandwidth anymore. Capacity is getting scarce and expensive too.
Why the memory wall hits inference harder than training
Training and inference put very different loads on the same hardware. Training processes large batches in parallel and keeps the processor busy. It’s compute-bound, which is exactly what GPUs are built for.
Inference splits into two phases. Prefill processes the whole input prompt in parallel and is compute-heavy, which also suits GPUs. Decode generates the output one token at a time, and every token requires another pass over the model weights and the KV cache. That’s where inference runs into the wall. (For how that split shapes hardware selection, see our guide to AI accelerators vs. GPUs.)
Adding compute without adding bandwidth doesn’t move the decode number—it just creates more idle silicon. In a Google Cloud Community write-up, Federico Iezzi measured a 96-GPU Nvidia B200 cluster on GKE running Qwen 3.5 27B. Tensor-core FLOPS utilization in decode sat at 4.4%, even while the cluster served about a million tokens per second.
The gap between the compute you’ve paid for and the compute decode can actually use is where cost and energy go. And it widens as context windows grow.
The KV cache problem
During prefill, the model computes and stores a key-value (KV) pair for every input token. That store is the KV cache—the model’s working memory of the conversation so far. During decode, the model reads the whole cache back to generate each new token, then appends an entry for the token it just produced.
The KV cache shares memory with the model weights. Longer contexts mean bigger caches, which leaves less room to batch requests together. Operators end up trading context length against concurrency: smaller batches give each request more bandwidth and return tokens faster, but they leave compute unused and spread the cost of reading the weights across fewer users. Cost per token goes up.
That’s why memory capacity, not just bandwidth, matters for inference at scale.
| Factor | Effect on memory |
|---|---|
| Larger models | More weight data to hold and read |
| Longer context windows | Larger KV cache per request |
| Higher concurrency | Higher bandwidth demand |
| KV cache growth | Competes with the weights for capacity |
How the industry is responding
Four approaches are in play. Each attacks a different part of the gap between how fast memory can deliver data and how fast compute can consume it.
HBM scaling
High-bandwidth memory (HBM) stacks DRAM dies vertically and connects them to the compute die through a silicon interposer. JEDEC’s HBM4 standard doubles the interface to 2,048 bits and specifies up to 2 TB/s per stack. With multiple stacks per package, the newest HBM4-equipped GPUs reach roughly 18 TB/s, compared to 3.35 TB/s on an Nvidia H100’s HBM3.
That’s real progress, but the architecture hasn’t changed. Memory and compute still sit on separate dies, and every weight and every cached key and value has to cross the interposer to reach compute. That crossing is what costs time and energy—roughly 2–3 pJ/bit for HBM4.
Supply is the other constraint. SK Hynix, Samsung, and Micron have reportedly pre-sold their entire 2026 HBM4 output to hyperscaler contracts.
On-chip SRAM
Some architectures skip off-chip memory and put SRAM—fast static memory built directly on the processor die—right next to compute. On-chip SRAM access takes on the order of a nanosecond, versus tens to hundreds of nanoseconds to reach DRAM, which makes it extremely attractive for low-latency decode.
The tradeoff is density. SRAM cells are roughly 10x the size of DRAM cells at about 100x the cost, so SRAM pools are fast but small. Wafer-scale designs answer that by building the largest chip possible, but holding a frontier model still means spreading weights across multiple wafers, and the interconnect between them is where cost and complexity land.
Digital In-Memory Compute (DIMC)
d-Matrix’s approach starts with SRAM and goes a step further. Rather than shuttling weights between separate memory and compute blocks, the DIMC architecture builds multiply-accumulate logic into the SRAM array itself, so the math happens where the weights already live.
[Graphic: d-Matrix “The AI Barrier” DIMC comparison, carried over from draft] Caption: Digital In-Memory Compute (DIMC) places the multiply inside the memory array, which shortens the path a weight travels to reach compute. Alt text: Side-by-side comparison of traditional and d-Matrix architectures. Left: traditional architecture with separate memory and compute blocks connected by a low-bandwidth HBM link. Right: d-Matrix Digital In-Memory Compute architecture with memory and multiply operations interleaved, achieving 150 TB/s per card.
The Corsair inference accelerator is built from chiplets—small dies combined into one package—which lets the design scale without a single massive monolithic die. Each card delivers 150 TB/s of Performance Memory bandwidth from 2 GB of on-chip SRAM, about 45x an H100’s HBM3, with 1,200 TB/s across an eight-card server. Because weights never cross an interposer, I/O energy drops to about 0.5 pJ/bit.
That’s the foundation of d-Matrix’s approach to breaking the memory wall for the memory-bound parts of inference. In a partner analysis combining measured and modeled data, Gimlet Labs ran a speculative decoding draft model for gpt-oss-120b on Corsairand reported 2–5x end-to-end request speedups over the same draft model on a GPU in interactivity-optimized configurations, and up to 10x in energy-optimized ones. Parasail began deploying Corsair alongside its Nvidia Hopper and Blackwell fleet in July 2026.
SRAM’s capacity limit still applies, though. That’s why Corsair is designed to work alongside GPUs rather than replace them—and why the next step is stacking memory.
3D stacked memory (3DIMC)
3DIMC bonds the compute die face to face with a DRAM die, connecting them vertically instead of across an interposer. The distance a weight travels shrinks from millimeters to microns, and the entire stacked surface becomes available for connections.
d-Matrix’s next-generation Raptor accelerator is built on 3DIMC. Pavehawk, our 3DIMC test silicon, is already validated in our labs at around 0.4 pJ/bit in worst-case conditions, and d-Matrix is targeting 10x the memory bandwidth and 10x the energy efficiency of HBM4. At rack scale, 144 Raptor accelerators form a memory pool of roughly 2.3 TB, enough to support models up to 3 trillion parameters in a single rack.
The point isn’t incremental gains. Models keep growing, and keeping up with them takes step-function improvements in memory.
What this means for infrastructure planning
The memory wall isn’t a future problem. It’s the constraint setting inference cost, latency, and scalability right now. Three planning considerations matter most:
- Memory bandwidth per watt. Facility power per rack, not the chip spec sheet, sets what a deployment can hold. Bandwidth that comes with a huge power draw doesn’t turn into throughput.
- A capacity scaling path. Model sizes should be able to grow without a proportional jump in power.
- A next-generation memory path. Your architecture should be able to adopt 3D stacked memory, and whatever comes after it, without a full platform redesign.
The most practical near-term answer is heterogeneous. A disaggregated approach matches each phase of inference to the hardware suited to it. GPUs handle compute-heavy prefill and attention. Memory-optimized accelerators take the memory-bound work: decode, draft models for speculative decoding, or the expert layers in attention-FFN disaggregation (AFD).
In AFD, the GPU keeps attention and its ever-growing KV cache while Corsair runs the fixed-size feed-forward network. Pairing one H100 with eight Corsair cards on GPT-OSS-20B cut time per output token from 2.59 ms to 1.28 ms at batch size 1.
This is complementary infrastructure, not a replacement. Offloading memory-bound work frees GPUs for the compute-heavy work they lead on.
How to evaluate memory-wall solutions
When you assess vendors and architectures, push past the spec sheet:
- What’s your effective memory bandwidth under real inference workloads—production models at production batch sizes, not peak spec?
- How does your architecture scale memory capacity without scaling power proportionally, and do you publish capacity and bandwidth at rack level?
- What’s your path to next-generation memory, and is it backed by silicon that’s already running?
The memory wall won’t be solved by adding more compute. It gets solved by designing around memory bandwidth.
Decode is where your fleet’s memory bandwidth runs out. d-Matrix Corsair runs alongside the GPUs you already own, and d-Matrix reports up to 10x faster interactive inference from that pairing. Contact us
FAQs about AI and the memory wall
Does the memory wall affect training and inference equally?
No. Training processes large batches in parallel, so it stays mostly compute-bound. Inference decode generates tokens one at a time and rereads the weights and KV cache for each, making it memory-bandwidth-bound. In one Google Cloud analysis, decode on 96 Nvidia B200 GPUs ran at just 4.4% tensor-core FLOPS utilization.
How does DIMC compare to HBM for inference?
HBM keeps memory on separate dies and moves data across an interposer. Digital In-Memory Compute performs the math inside on-chip SRAM, so weights never leave the memory array. d-Matrix reports 150 TB/s per Corsair card versus roughly 18 TB/s for HBM4-equipped GPUs, at a fraction of the energy per bit.
What does 3DIMC mean for future AI chip design?
3DIMC bonds compute and DRAM dies face to face, combining bandwidth closer to SRAM with the capacity of DRAM. That removes the forced choice between fast-but-small and big-but-slow memory. d-Matrix’s Raptor accelerator is built on 3DIMC and targets 10x the bandwidth and energy efficiency of HBM4.
Will the memory wall get worse as models grow?
On current trajectories, yes. Compute keeps scaling faster than memory bandwidth, frontier models have passed a trillion parameters, and context windows now reach a million tokens. Each trend increases the data that moves per token, which is why memory architecture, not raw compute, will set the pace of AI infrastructure.
Can you solve the memory wall by adding more GPUs?
Not efficiently. More GPUs add bandwidth, but replicated models each read their own copy of the weights, and tensor parallelism adds network hops between every layer. Per-token decode time doesn’t fall in proportion to the hardware you add, so cost rises faster than performance.
