You added more GPUs and the numbers barely moved. Latency still spikes under load, cost per token won’t come down, and the dashboard says your chips are busy while throughput sits well below what the spec sheet promised.
The instinct is to throw more compute at it—bigger GPUs, more of them. For large-model inference, that usually solves the wrong problem.
The real constraint in most production inference is memory bandwidth. Every token a model generates requires pulling weights and cached state out of memory, and that trip—not the math—sets the pace. More FLOPS don’t help much when the chip spends most of its time waiting on data.
Here’s where the AI inference bottleneck actually forms, why faster GPUs deliver diminishing returns against it, and how to tell whether your own deployment is memory-bound.
Key takeaways
- For large-model inference, the dominant bottleneck is memory bandwidth, not compute. Decode spends most of its time moving data rather than doing math.
- Standard GPU utilization metrics can show chips as fully busy while the compute units are mostly waiting on memory.
- Compute has grown far faster than memory bandwidth for years—a gap the industry calls the memory wall.
- Tail latency (p99), not average latency, governs production SLAs, and memory pressure hits the tail first.
- Memory-optimized accelerators attack data movement directly and work alongside existing GPU fleets rather than replacing them.
Slow, costly inference is usually a memory problem. d-Matrix builds memory-optimized inference accelerators that work alongside your existing Nvidia GPUs and take over the memory-bound work, so you get more out of the hardware you already run. Contact us
What is the AI inference bottleneck?
The AI inference bottleneck is whatever single resource caps throughput and latency when you serve a model, no matter how much of everything else you add. Getting the diagnosis right matters, because the wrong one leads to spending that changes nothing.
Training and inference stress hardware differently. Training runs huge batches of dense matrix math, which is exactly what GPUs are built for. Inference serves real users waiting on a response, which makes it latency-sensitive—and for large language models, how fast you can generate each token is usually limited by how fast the hardware can move data.
Inference also splits into two phases that behave very differently. Prefill processes your entire prompt at once. It’s compute-heavy and highly parallel, which makes it a great fit for GPUs. Decode then generates the response one token at a time, with each token depending on the one before it. It’s memory-bound and latency-sensitive, and for most interactive workloads it’s the phase users actually feel.
Every decode step reads the model’s weights and its accumulated context out of memory before any math happens. When those reads are the slow part, faster compute just sits idle.
Why it isn’t a compute problem
Adding compute to a memory-bound workload is like adding lanes to a highway that empties onto a one-lane bridge. The extra lanes don’t help, because the constraint is downstream.
A few definitions make the mechanism concrete. FLOPS (floating-point operations per second) measures a chip’s raw compute throughput—how fast it can multiply and add. HBM (high-bandwidth memory) is the DRAM packaged alongside a GPU that holds model weights during inference. The KV cache (key-value cache) is the attention state from every token processed so far, which the model re-reads to produce the next one.
During decode, three kinds of data compete for the same memory bandwidth on every token:
- Weights, the model parameters needed to compute the next token. Mixture-of-experts (MoE) models only activate a subset of experts per token, but which ones changes from token to token—and at larger batch sizes, nearly all of them end up doing work on every step.
- Activations, the intermediate values produced as data flows through each layer.
- The KV cache, which grows with context length and batch size.
When memory reads take longer than the calculations, compute units wait. That’s the GPU memory bottleneck in a nutshell: the compute is there, but the memory system can’t feed it fast enough. It’s also why “HBM bottleneck” has become shorthand for the whole problem—HBM’s bandwidth, not the cores, sets the ceiling.
You can see it in the utilization numbers if you look closely. The standard GPU utilization metric only tracks whether a kernel is running, not whether the compute units are doing useful math. In a Google Cloud Community write-up, Federico Iezzi measured a 96-GPU Nvidia B200 cluster running Qwen 3.5 27B. Tensor-core FLOPS utilization in decode sat at 4.4% while the cluster served about a million tokens per second. The hardware was there. It was mostly waiting.
The memory bandwidth bottleneck, explained
Decode speed comes down to how fast the system can move data between memory and the processor—and that rate hasn’t kept up with compute for a long time.
Researchers at UC Berkeley found that peak hardware compute has scaled roughly 3x every two years, while DRAM bandwidth has scaled only about 1.6x over the same period. Every hardware generation adds compute faster than it adds the bandwidth to feed it, so a memory-bound workload can use a smaller share of the chip with every release. That widening gap is the memory wall, and it’s now the defining constraint of large-model serving.
The wall hits hardest during the memory-bound decode phase. Prefill can amortize each memory read across an entire prompt processed in parallel. Decode can’t: it generates one token at a time, and each token forces another pass over the weights and a KV cache that keeps getting bigger.
An Nvidia H100 streams roughly 3.35 TB/s out of its HBM3, and the newest HBM4 reaches about 18 TB/s. Those are enormous numbers. Decode still waits on them, because it’s making that round trip for every single token.
More FLOPS can’t fix a bandwidth shortfall. A different memory architecture can.
Why tail latency governs production SLAs
Averages hide the requests that break your service-level agreements. In production, p99 latency—the response time of the slowest 1% of requests—matters more than the median. A fast average with a slow tail still means a meaningful share of your users are waiting too long.
Two metrics tell you what that tail is made of. TTFT (time to first token) is how long a user waits for the first token to appear, and it’s driven mostly by prefill. TPOT (time per output token) is the pace of every token after that, driven by decode. The memory bottleneck shows up most sharply in TPOT under load, where variance stacks up from a few places:
- Batching queues. Dynamic batching holds requests while they’re grouped, and queue depth spikes when traffic arrives in bursts.
- KV cache pressure. When the KV cache outgrows available memory, the serving system has to evict and recompute state—right when it’s busiest.
- Thermal throttling. Sustained load drives temperatures up, and chips drop clocks to stay inside their thermal envelope, slowing the tail further.
As an illustrative framing rather than a measured benchmark, p99 latency can run roughly 2–5x higher than the median under real production load. The exact multiple depends on your workload and infrastructure, but the direction is consistent: the tail degrades faster than the average as memory pressure rises.
How architecture solves what more hardware can’t
[Graphic: Corsair rack-scale spine/leaf topology, carried over from draft]
If the bottleneck is data movement, the fix has to change how data moves—not how fast the math runs. That’s the premise behind a memory-centric inference platform: hardware designed around memory bandwidth rather than peak FLOPS.
The d-Matrix platform is built on Digital In-Memory Compute (DIMC), which integrates compute and memory on the same silicon. Rather than shuttling weights back and forth between separate memory and processor chips, Corsair keeps them in on-chip SRAM directly adjacent to the compute. Each Corsair card carries 2 GB of SRAM with 150 TB/s of memory bandwidth—about 45x the HBM3 on an H100—at 0.5 pJ/bit of I/O energy compared to 3–5 pJ/bit for HBM. An eight-card server brings 16 GB of SRAM and 1,200 TB/s of aggregate bandwidth.
SRAM is extremely fast, but it isn’t huge. So rather than replacing GPUs, Corsair works alongside them in disaggregated pipelines, where each part of inference runs on the hardware best suited to it. GPUs keep the compute-heavy work they’re built for—prefill, attention, and the growing KV cache. Corsair takes the memory-bound work: decode, draft models for speculative decoding, or the expert layers in attention-FFN disaggregation.
The results are measurable. Running GPT-OSS-20B with attention on a single Nvidia H100 and the expert layers on eight Corsair cards cuts time per output token from 2.59 ms to 1.28 ms at batch size 1—roughly half. In production, Parasail is deploying Corsair alongside its Nvidia Hopper and Blackwell fleet, pairing GPUs for compute-intensive prefill with Corsair for latency-sensitive decode. d-Matrix reports up to 10x faster token generation with that approach.
The same model carries into the next generation. d-Matrix is partnering with Nvidia on NVLink Fusion to bring Raptor, our 3D-DRAM accelerator, into the same tray-based rack architecture as Vera Rubin. Raptor racks scale up over NVLink and connect to GPU racks for heterogeneous inference with Nvidia, with stacked DRAM pushing past SRAM’s capacity limits for frontier-scale models.
How to diagnose your own inference bottleneck
Before you buy more hardware, find out whether more hardware would actually help. Track these four signals under realistic, sustained load—not a quiet benchmark:
- Tokens per second is your real throughput ceiling, measured while serving concurrent traffic.
- p99 latency, split into TTFT and TPOT, tells you whether prefill or decode is the slow phase.
- Memory-bandwidth utilization is the share of available bandwidth in use. If it’s pinned high while compute utilization sits low, you’re memory-bound.
- Cost and power per token—total cost of ownership (TCO) per token and watts per token—tell you what each unit of output actually costs to serve.
The tell is a familiar pattern: the GPU looks busy, throughput has plateaued, and memory bandwidth is saturated. Adding FLOPS won’t help, because compute isn’t the constraint.
When you talk to vendors, ask questions that surface the same distinction:
- Is my workload memory-bound or compute-bound, and how are you measuring that?
- What memory architecture does your hardware use, and how much bandwidth reaches the decode phase?
- Does your solution work alongside my existing GPU fleet, or does it require replacing it?
Those answers separate hardware that adds lanes to the highway from hardware that widens the bridge.
Faster GPUs won’t fix a decode phase that’s waiting on memory. d-Matrix pairs memory-optimized Corsair accelerators with your existing Nvidia fleet to take on memory-bound work, so you raise utilization, cut cost per token, and hit your latency targets without replacing the hardware you have. Contact us
FAQs about the AI inference bottleneck
Is AI inference limited by memory or compute?
For large language models, it’s usually memory. Decode reads the model’s weights and KV cache for every token it generates, so memory bandwidth—not raw compute—sets the pace. Prefill, which processes the whole prompt in parallel, is the more compute-bound phase.
What is the memory wall?
The memory wall is the growing gap between how fast processors compute and how fast data moves to and from memory. Peak compute has grown roughly 3x every two years versus about 1.6x for memory bandwidth, so memory-bound workloads can use a shrinking share of each new chip.
Will faster GPUs fix inference latency?
Only if compute is your bottleneck. A faster GPU helps compute-bound prefill, but may do little for decode when memory bandwidth is the constraint. Measure TTFT and TPOT separately before you upgrade.
What is HBM and why does it bottleneck inference?
HBM (high-bandwidth memory) is the DRAM packaged alongside a GPU that holds model weights and the KV cache. During decode, repeated reads of that data saturate its bandwidth, so token generation stalls even when the compute units have capacity to spare.
How do I tell if my deployment is memory-bound?
Measure memory-bandwidth utilization, compute utilization, and throughput under sustained load. If bandwidth is saturated while compute stays low and throughput has plateaued, you’re memory-bound. Splitting latency into TTFT and TPOT shows which phase is the slow one.
