Your reasoning agent just chained six tool calls, each one waiting on the last. Every millisecond of token generation latency compounds across that chain, and your user is waiting on all of it. Low latency AI inference describes systems that generate model outputs fast enough that whatever consumes them—an orchestration layer, an agentic pipeline, or a person watching a chat window—experiences no perceptible delay.
That bar keeps getting harder to clear. A single user request can now set off dozens of inferences: additional model calls, tool use, retrieval steps, and separate modalities like speech generation. The user only sees the final answer, but the latency they feel is the sum of every step inside it, from frontier model calls down to the small models in between.
Meanwhile, the standard way to make serving economical—batching more requests together—makes each individual request slower. Here’s why speed and scale pull against each other, and how to get both.
Key takeaways
- Low latency AI inference means generating outputs fast enough that downstream applications and users see no perceptible delay. What counts as “low” depends on the workload.
- Agentic and reasoning workloads compound latency across steps, and every thinking token is still a token generated during decode.
- Decode is memory-bandwidth-bound, not compute-bound. That’s the memory wall, and it’s the core reason low latency is hard.
- Batching improves throughput economics but slows every request in the batch.
- Memory-optimized accelerators like d-Matrix Corsair put far more bandwidth behind decode and run alongside existing GPU fleets rather than replacing them.
Agentic chains pay for latency at every step. d-Matrix accelerators take on memory-bound decode alongside your existing Nvidia GPUs, so you get more out of the fleet you already run. Contact us
What is low latency AI inference?
Latency in AI inference is the time between sending a request and receiving output tokens. Two metrics define it:
- Time to first token (TTFT) is how long a user waits for the first token to appear. It’s driven by prefill, the phase that processes the whole prompt at once.
- Time per output token (TPOT) is how long it takes to generate each subsequent token during decode. It’s what a user experiences as speed while a response streams onto the screen.
Latency and throughput aren’t the same thing. Throughput is how many tokens your system generates across all requests, while latency is how fast each individual request responds. On most hardware, optimizing one degrades the other: a smaller batch gives each request more memory bandwidth and returns tokens sooner, but it leaves compute capacity unused.
Why low latency matters now
Agentic AI, reasoning models, and real-time applications all generate far more tokens per user request than a single chat turn, so any per-token delay gets paid many times over.
- Agentic pipelines chain inference calls in sequence, with each step waiting on the one before it. Delay accumulates across the chain while the user waits on the total.
- Reasoning models generate long sequences of thinking tokens before producing an answer. Each thinking token is still a decode step with the same per-token cost.
- Real-time applications like coding assistants and conversational agents are consumed as the output streams, so each token’s delay reaches the user directly.
Why low latency AI inference is hard
During decode, generative models are memory-bandwidth-bound, not compute-bound. For every output token, the processor reads model weights and the KV cache—the keys and values stored for every earlier token, which grows as the response gets longer.
Frontier models make this harder. Open-weight models like DeepSeek-V4-Pro and Qwen3.8-Max now run past a trillion parameters. Mixture-of-experts architectures activate only a fraction of those per token, but the full model still has to live in memory. On a GPU, that memory is HBM (high-bandwidth memory): DRAM stacks on dies physically separate from the compute. Every read crosses that gap, and that link sets the per-token ceiling.
It shows up in the utilization numbers. In a Google Cloud Community write-up, Federico Iezzi recorded 4.4% tensor-core FLOPS utilization during decode on 96 Nvidia B200 GPUs serving Qwen 3.5 27B, even as the cluster returned roughly a million tokens per second. Decode spent most of its time waiting on memory rather than computing—and that wait is exactly what users experience as TPOT.
This is the memory wall: compute has scaled far faster than memory bandwidth for decades, so adding compute doesn’t shorten the wait. (For how prefill and decode split across different hardware, see AI accelerator vs GPU.)
The two bottlenecks: bandwidth and capacity
Low latency inference runs into two different memory constraints:
- Bandwidth determines how fast the processor can read the weights and KV cache for each output token. The JEDEC HBM4 standard specifies up to 2 TB/s per stack, and the newest HBM4-equipped GPUs reach roughly 18 TB/s in total.
- Capacity determines how much of a model sits close to the compute, on a card and across a rack. Adding DRAM raises how much fits, but not how fast the processor can read it.
Corsair handles the two separately. Its Performance Memory—on-chip SRAM—is built for bandwidth, while its Capacity Memory, a larger DRAM tier, holds what doesn’t need to be read at full speed on every step. In our attention-FFN disaggregation work, for example, the LM head streams in from DRAM while Corsair finishes the remaining layers, keeping that off-chip traffic essentially hidden. For where the capacity side goes next, see our look at scaling AI inference with 3DIMC.
The batching tradeoff: latency vs. throughput
Batch size sets the economics of a GPU-only inference fleet, and it puts throughput and latency directly at odds.
Larger batches share a single read of the model weights across more requests, which raises utilization and cuts cost per token. But every request in the batch waits on the slowest work in it, so per-token latency climbs as the batch grows. Smaller batches return tokens faster and leave expensive compute sitting idle.
Continuous batching—adding and removing requests from a running batch instead of waiting for the whole batch to finish—recovers some of the loss. But it doesn’t change the memory bandwidth available to each request.
Achieving low latency AI inference at scale means solving that tradeoff: delivering per-user interactivity without giving up the throughput economics that make serving commercially viable.
How to achieve low latency AI inference without sacrificing scale
The tradeoff exists because the memory link can’t feed every request in a batch at once. An Nvidia H100 reads from HBM3 at 3.35 TB/s and a B200 at about 8 TB/s, so every request added to a batch waits longer for its share. The way out is to put far more bandwidth behind decode.
DIMC puts compute inside the memory array
d-Matrix’s DIMC architecture (Digital In-Memory Compute) builds compute logic directly into on-chip SRAM, so the math happens next to the data rather than after a round trip across HBM. Shortening that path is what produces the bandwidth: the Corsair inference accelerator delivers 150 TB/s of Performance Memory bandwidth per card, 300 TB/s per dual card, and 1,200 TB/s across an eight-card server.
Data movement doesn’t disappear entirely, and SRAM holds far less per card than an HBM stack. That’s why Corsair is designed to work alongside GPUs rather than replace them.
Disaggregation puts each phase on the right hardware
A disaggregated approach splits inference across hardware. GPUs handle compute-dense prefill, measured by TTFT, while Corsair handles bandwidth-bound decode and other memory-bound work, measured by TPOT.
That split is a tradeoff of its own. It adds a hop between phases, a scheduler that has to keep both pools busy, and data transfers between them. d-Matrix’s JetStream NIC keeps that hop small: devices write data directly into each other’s memory with no host CPU involvement, and each pipeline transfer adds less than 2 µs. Operators take on the extra complexity because decode is where most tokens are generated and most per-token cost accumulates.
The payoff shows up in TPOT. In an attention-FFN disaggregation configuration, one Nvidia H100 paired with eight Corsair cards serves GPT-OSS-20B at 845 tokens per second per user, cutting TPOT from 2.59 ms on the H100 alone to 1.28 ms. Scale that to 64 concurrent users with two H100s sharing the same eight cards, and the system completes each decode step in 2.46 ms while delivering 20,413 tokens per second in total.
Corsair in production
Heterogeneous inference is already running at commercial scale. Parasail is deploying Corsair alongside its Nvidia Hopper and Blackwell fleet, with GPUs on compute-intensive prefill and Corsair on latency-sensitive decode, and d-Matrix reports up to 10x faster interactive inference from the pairing.
Gimlet Labs, which runs an inference cloud for agentic workloads, tested speculative decoding on Corsair, running a small draft model that proposes tokens for a larger model to verify in batches. Serving GPT-OSS-120B with a 1.6B-parameter draft model at 8K input and 1K output tokens, Gimlet reported 2–5x end-to-end request speedups over the same draft model on a GPU in interactivity-optimized configurations, using a mix of measured and modeled data.
Corsair is also built to fit where you already are. It’s an air-cooled, full-height PCIe Gen5 card that drops into existing servers. That matters given the scale of what’s being built: McKinsey projects $6.7 trillion in global data center capital expenditure by 2030, $5.2 trillion of it driven by AI, much of it in facilities designed around standard server formats.
Bandwidth only helps if the serving software can use it. d-Matrix’s Aviator software stack is built on open software including PyTorch, MLIR, and Triton. It handles model compression through post-training quantization and quantization-aware training, and its distributed inference engine supports low-latency serving across multi-card deployments.
How to get started
Start by profiling your current TPOT and end-to-end response latency at your real concurrent-user count, batch sizes, and context lengths. That tells you where your deployment sits between latency and throughput today.
If latency degrades as batch sizes grow, the constraint is memory bandwidth, not serving-stack tuning—no amount of batching strategy or framework optimization changes the bytes per second available to each request. From there, pilot a heterogeneous configuration on a single latency-sensitive workload, benchmark it on your own models, and expand based on what you measure.
Your decode phase is limited by the bandwidth behind it. d-Matrix Corsair accelerators put far more bandwidth behind decode, alongside your existing GPUs, and change what the latency-throughput tradeoff costs you. Contact us
FAQs about low latency AI inference
What latency is considered “low” for AI inference?
It depends on the workload. Agentic pipelines push the bar toward hundreds or thousands of tokens per second per user, because every step waits on the one before it. For a single interactive stream, d-Matrix works toward a single-digit-millisecond time per output token—a working target, not an industry standard.
How does low latency differ from high throughput?
Low latency means each request returns tokens quickly, measured as TPOT during decode. High throughput means serving many requests at once. In GPU-only serving they conflict: batching more requests raises throughput and lengthens every request’s wait. More memory bandwidth behind decode raises the batch size at which that conflict starts to bite.
Can you achieve low latency at high batch sizes?
It depends on the bandwidth behind the batch. Larger batches mean more memory traffic per decode step, and a limited memory link makes each request wait its turn. Architectures that put far more bandwidth behind decode, like the 150 TB/s of Performance Memory bandwidth on a single Corsair card, push that limit much further out.
What is time per output token, and why does it matter?
Time per output token (TPOT) is how long the system takes to generate each token after the first, and it’s driven by the memory-bound decode phase. It sets how fast responses stream once they start, and it compounds: for a 500-token response, shaving 5 ms off TPOT saves 2.5 seconds.
