Inference now costs more than training. Gartner forecasts that inference will account for 55% of AI-optimized cloud infrastructure spending in 2026—$23.3 billion against $19 billion for training—rising to 59% in 2027.
That shift changes the math. Training is a cost you pay while building a model. Inference is a cost you pay for as long as the model is in production, and it scales with every user, every agent, and every token.
Here’s the uncomfortable part: much of that spend buys accelerator capacity that isn’t producing anything. During decode, the phase that generates output one token at a time, accelerators spend far more time waiting on memory than computing—and you’re paying for every idle cycle.
Here’s where inference cost actually comes from, which levers reduce it fastest, and why infrastructure architecture determines your long-term cost per token.
Key takeaways
- Inference is now the larger share of AI cloud infrastructure spending. Gartner forecasts 55% in 2026 and 59% in 2027.
- Utilization drives cost per token. Decode is bound by memory bandwidth, so compute you’ve already paid for sits idle.
- Power and cooling are part of total cost of ownership, not add-ons. Shortening the wait on memory lowers energy per token.
- Software levers—right-sizing, quantization, batching, and caching—deliver savings in days, but they don’t change what constrains the hardware.
- Durable savings come from architecture. Matching each phase of inference to the right hardware changes the cost structure instead of trimming it.
Inference cost is an infrastructure problem before it’s a model problem. d-Matrix accelerators work alongside your existing Nvidia GPUs, offloading memory-bound work to silicon built for inference economics. Contact us
Where AI inference costs actually come from
Inference spend usually shows up as an API bill or a cloud invoice. Inside the data center, three things set that number: the accelerator hardware, the power and cooling that keep it running, and how much of that hardware is doing useful work at any given moment.
Hardware is the visible line item. AI accelerators are capital-intensive, and the highest-end parts come with premium pricing and long procurement cycles. Power and cooling are the recurring half of the same total cost of ownership (TCO)—the part you pay every month.
Utilization is what turns that total into a cost per token. You pay for the whole accelerator whether it’s doing useful work 90% of the time or 4% of the time, so the same TCO can buy wildly different amounts of output. That’s why cheaper hardware alone doesn’t fix the arithmetic. Lower AI inference cost comes from raising utilization, lowering energy per token, and matching hardware to the phase of inference it runs.
The memory bandwidth tax
Inference runs in two phases. Prefill processes the whole prompt in one pass and is compute-heavy, which GPUs handle well. Decode generates output one token at a time and is bound by how fast memory can deliver data, not how fast the chip can compute. (For the full breakdown, see AI accelerator vs GPU.)
Every decode step reads model weights out of memory, along with the KV cache—the keys and values stored for every earlier token in the sequence. The weight traffic repeats on every token, and the cache traffic grows with each one generated, so long responses get more expensive per token as they go.
Moving that data takes far longer than the math it feeds. In a Google Cloud Community analysis of Qwen 3.5 27B running across 96 Nvidia B200 GPUs, tensor-core FLOPS utilization during decode came in at just 4.4%. That’s the memory bandwidth tax: compute you bought, waiting on data.
And it’s getting worse, not better. Gholami and co-authors found that compute has scaled roughly 3x every two years against about 1.6x for memory bandwidth—the widening gap known as the memory wall. Each hardware generation spends proportionally more of every decode step waiting.
The cost of power and cooling
Two things draw power in an inference deployment: the accelerator itself, and the cooling system that removes the heat it produces.
The highest-power GPUs now draw more than a kilowatt per card and need liquid cooling, which most existing data center halls aren’t plumbed for. That turns cooling into a facilities decision measured in years of construction, not months.
Corsair takes a different approach. It’s air-cooled and ships as a standard PCIe card, so it fits the slots servers already have and the data center halls built before liquid cooling.
Idle time costs energy, too. An accelerator drawing power while it waits on memory still consumes energy but produces no tokens. Shorten that wait, and the same power draw covers more finished work.
Software levers that lower cost fast
Software changes land in days and don’t need new hardware. Right-sizing, quantization, batching, and caching each reduce the work an accelerator has to do per request, which lowers what each request costs. What they don’t change is the constraint underneath: decode still waits on memory.
Model right-sizing and quantization
Right-sizing is the single highest-impact lever: run the smallest model that still clears your quality bar. Serving a 70B model for a task a fine-tuned 7B model handles means paying for capability the workload never touches—and in agentic pipelines, plenty of individual steps don’t need a frontier model at all.
Quantization attacks the same bill from a different angle by storing weights at lower numerical precision. Most models ship in FP16 or BF16, two bytes per weight. Moving to 8-bit halves the memory footprint, and 4-bit quarters it: a 27B-parameter model drops from about 54 GB at FP16 to about 27 GB at 8-bit and 13.5 GB at 4-bit. Because decode is bound by memory traffic, fewer bytes per weight means less data to move on every token.
Post-training quantization (PTQ) is the fastest path, and modern methods like GPTQ, AWQ, and FP8 usually hold accuracy close to the original model, though the drop is never exactly zero. Quantization also determines what fits where. At 4-bit precision, all of GPT-OSS-20B’s experts fit in the 16 GB of on-chip SRAM across a single eight-card Corsair server. For production workloads with stable quality bars, distillation—training a smaller model on a larger model’s outputs—can push savings further.
Batching and caching
Batching groups concurrent requests so a single pass through the model serves all of them. The cost of reading the weights is paid once per batch instead of once per request, so total tokens per second goes up.
The catch is latency. Every request in a bigger batch waits on the slowest work in it, while smaller batches return tokens faster but leave compute idle. On a single architecture, the right batch size is whatever your latency target allows—which is why teams serving high-concurrency traffic increasingly pair batching with hardware built for low-latency batched throughput rather than choosing between the two.
Caching removes work instead of spreading it. Prefix caching stores the KV cache for a shared prompt prefix, so prefill doesn’t rebuild it for every request that starts the same way. Long system prompts and shared document context benefit most. Semantic caching goes further, answering repeated or near-duplicate queries without running the model at all, and prompt compression trims the input tokens you process on every call.
Infrastructure architecture is the structural cost lever
Every software lever reduces the work a request needs. None of them changes the ratio of compute capacity to memory bandwidth in your hardware—and that ratio is what produced the 4.4% utilization figure. Moving that number means changing what decode runs on. That’s the difference between lowering cost per request and lowering cost per token.
Purpose-built inference hardware
GPUs are extraordinary at prefill, which reuses each weight across many tokens at once. Decode flips that, reading weights again for every token it produces. Running decode on a GPU means paying for compute the phase can’t use, while the memory bandwidth it does need stays the binding limit—about 8 TB/s on an Nvidia B200, and roughly 18 TB/s on the newest HBM4-equipped GPUs.
Inference-specific architectures start from that limit instead of inheriting it. The Corsair inference accelerator is built on d-Matrix’s Digital In-Memory Compute (DIMC) architecture, which integrates compute into on-chip SRAM so weights never cross an HBM link. Each card delivers 150 TB/s of memory bandwidth, and an eight-card server reaches 1,200 TB/s.
Heterogeneous and disaggregated inference
You don’t have to replace a GPU fleet to get there. A disaggregated inference pipeline runs each phase on the hardware that suits it: prefill stays on GPUs, memory-bound work moves to a memory-optimized accelerator, and the two hand data between them.
Heterogeneous deployments are already in production. Parasail is pairing its Nvidia Hopper and Blackwell GPUs with Corsair—GPUs on prefill, Corsair on decode—and d-Matrix reports up to 10x faster interactive inference from that configuration. In an attention-FFN disaggregation setup, pairing one H100 with eight Corsair cards on GPT-OSS-20B cut time per output token roughly in half, from 2.59 ms to 1.28 ms.
That’s the point: faster tokens from the GPUs you already own. More of the hardware you’ve paid for produces output, and cost per token falls with it.
Building an inference cost optimization roadmap
Start with measurement. Instrument your serving stack to attribute cost by phase, by model, and by utilization, so you know whether prefill or decode is where your spend concentrates before you change anything.
Then sequence the work by time-to-value:
- Software first (days). Right-size models, quantize, and adopt continuous batching and prefix caching.
- Infrastructure next (quarters). Match hardware to workloads and move toward disaggregated, heterogeneous serving.
- Take the hybrid path. Most organizations won’t replace their GPU fleet overnight, and they don’t need to. Offload decode-heavy workloads to memory-optimized accelerators first, benchmark on your own models, measure the savings, and expand from there.
When you’re ready to scale, JetStream and SquadRack extend the same pipeline to rack scale over standard PCIe and Ethernet.
The savings that last come from utilization and hardware fit, not from squeezing another few percent out of a batch size.
Where decode runs is what sets your cost per token. d-Matrix complements your Nvidia fleet by offloading memory-bound decode to memory-optimized hardware, so the capacity you already pay for turns into output. Contact us
FAQs about inference cost
How do I calculate inference cost per token?
Divide the fully loaded cost of your serving infrastructure over a period—hardware amortization, power, cooling, and operations—by the total tokens generated in that period. If accelerators sit at low utilization during decode, your effective cost per token is far higher than the hardware’s sticker price suggests.
When does self-hosted inference beat API pricing?
When your sustained token volume is high enough to keep accelerators busy. APIs cost less at low or bursty volume because you only pay for tokens used, not for hardware idling between requests. Utilization decides the break-even, not the accelerator’s price, so model your real traffic pattern, including idle time.
How does quantization affect output quality?
Modern methods like GPTQ, AWQ, and FP8 usually preserve accuracy close to the original model while cutting the memory footprint in half at 8-bit and in half again at 4-bit. The drop is never exactly zero, so test quantized models against your own quality metrics. Quantization-aware training can recover some of the difference.
Does purpose-built inference hardware work with existing ML pipelines?
Yes, though expect a porting and validation step. Inference accelerators are designed to sit inside an existing serving stack rather than replace it. d-Matrix’s Aviator software stack is built on open software including PyTorch, MLIR, and Triton, and supports standard quantization (PTQ and QAT) and compression workflows.
