In September 2026, d-Matrix and Nvidia announced a multi-year collaboration to bring d-Matrix inference accelerators into Nvidia’s MGX rack-scale architecture through NVLink Fusion. Two months earlier, Parasail began deploying d-Matrix Corsair accelerators alongside its Nvidia Hopper and Blackwell fleet in production.
Neither of those is a story about an accelerator beating a GPU. Both are about the two working together.
“AI accelerator vs GPU” is still how most people start researching the category, and it’s a fair place to start. But it’s the wrong comparison. Generative inference runs in two phases with opposite hardware appetites—one compute-hungry and parallel, the other memory-hungry and sequential. GPUs are built for the first. Memory-optimized accelerators are built for the second. Forcing either one to do both means accepting a compromise you don’t have to make.
Here’s what each architecture does best, why pairing them outperforms either one alone, and what to look for when you add an accelerator to a GPU fleet.
Key takeaways
- AI accelerators are purpose-built for a specific workload, which today mostly means inference. GPUs are general-purpose and run training, prefill, batch serving, and interactive decode on the same part.
- The core architectural difference is where memory sits. Memory-optimized accelerators keep model weights on-chip next to compute, while GPUs stream them from high-bandwidth memory (HBM) on separate dies.
- Inference has two phases. Prefill is compute-bound and suits GPUs. Decode is memory-bandwidth-bound and suits memory-optimized accelerators.
- The real question isn’t which architecture wins, but which one fits which phase. Heterogeneous inference runs each phase on the hardware suited to it.
- It’s already happening: Nvidia and d-Matrix are integrating at rack scale through NVLink Fusion, and Parasail runs Corsair beside its Hopper and Blackwell GPUs.
You don’t have to pick one architecture for your whole pipeline. d-Matrix builds inference accelerators that run alongside your GPUs rather than replacing them, so your existing fleet keeps doing the work it’s best at. Contact us
What is an AI accelerator?
An AI accelerator is a processor purpose-built to run a specific class of AI workload as efficiently as possible. For most of the current generation of accelerators, that workload is AI inference—serving a trained model to live users and generating its output.
GPUs took a different path. They started in graphics, where rendering means running enormous numbers of matrix operations in parallel. Deep learning turned out to be built on the same math, so GPUs became the default for training and, by extension, inference. Today a single GPU runs training, prefill, batch serving, and interactive decode on the same silicon.
That breadth is a real strength, but it comes with tradeoffs. Generating output token by token is a different shape of problem from training. Each token waits on the one before it, so any delay in one step gets paid again on every step after it. A 500-token answer pays that delay 500 times, and the user watches the text arrive at exactly that pace.
An accelerator designed around that pattern can make architectural choices a general-purpose part can’t—most importantly, about where the memory goes.
How AI accelerators differ from GPUs architecturally
The clearest difference is where the model lives while it runs.
On a GPU, model weights sit in HBM—stacks of DRAM packaged next to the compute die and connected through a silicon interposer. HBM keeps getting faster: the JEDEC HBM4 standard specifies up to 2 TB/s per stack, and the newest HBM4-equipped GPUs reach roughly 18 TB/s, up from 3.35 TB/s on an Nvidia H100. But memory and compute still sit on separate dies, and every weight has to cross that gap before it can be used.
Memory-optimized accelerators put the memory on the chip, directly next to the compute. d-Matrix Corsair is built on a DIMC architecture (Digital In-Memory Compute) that integrates compute into on-chip SRAM. Each card delivers 150 TB/s of memory bandwidth from 2 GB of SRAM—about 45x an H100—and an eight-card server reaches 1,200 TB/s.
That gap matters because of the memory wall: compute has scaled far faster than memory bandwidth for decades, so adding FLOPS does little for memory-bound work. What decides performance is how fast data reaches the compute.
The tradeoff runs the other way, too. On-chip SRAM is extremely fast but small, while HBM holds far more data. That’s exactly why the two architectures complement each other rather than compete.
Why the “vs” framing misses the point
No single architecture serves every workload optimally. A chip tuned for one phase of inference is a compromise for the others.
Heterogeneous computing combines different types of processors in the same pipeline, each handling the work it does best. It’s the same principle that put CPUs and GPUs in the same server years ago, now applied to the phases of AI inference. The question isn’t which architecture wins outright. It’s which one fits which phase.
The prefill/decode split
Every generative inference request runs in two phases.
Prefill processes the entire input prompt at once. The model reads every input token in parallel and builds the KV cache—the keys and values it stores for each token so later steps don’t have to recompute them. Prefill is compute-bound and highly parallel, which is exactly what GPUs are built for. It determines time to first token (TTFT), or how long a user waits before any output appears.
Decode generates the output one token at a time, each depending on the one before it. Every step reads the model weights and the full KV cache, which grows with every token generated. (Mixture-of-experts models activate only some of their weights per token, but at production batch sizes most experts end up in play on every step.) Memory bandwidth sets the pace, and time per output token (TPOT) is what a user experiences as speed.
Run both phases on one architecture and batch size becomes your main lever—and it pulls throughput and latency in opposite directions. Bigger batches group more requests into each pass and raise total tokens per second, but every request in the batch waits on the slowest work in it. Smaller batches feel snappier but leave expensive compute sitting idle.
You can see how lopsided decode is on a GPU. In a Google Cloud Community write-up, Federico Iezzi measured 4.4% tensor-core FLOPS utilization during decode while serving Qwen 3.5 27B on 96 Nvidia B200 GPUs, even as the cluster pushed about one million tokens per second. Nothing was broken. That’s simply what a bandwidth-bound phase looks like on hardware built for compute-bound work.
How accelerators and GPUs work together in practice
A disaggregated approach splits the inference pipeline across both architectures. Each phase runs on the hardware that suits it, and the two exchange data over a fast interconnect. GPUs take the compute-heavy work, memory-optimized accelerators take the memory-bound work, and the line between them can be drawn in more than one place:
- Prefill/decode disaggregation. GPUs handle prefill, and accelerators handle decode.
- Speculative decoding. An accelerator runs a small draft model that proposes tokens in rapid succession, and the GPU verifies them in a single parallel pass.
- Attention-FFN disaggregation (AFD). The GPU handles attention and the growing KV cache, while the accelerator runs the feed-forward network—where most of a model’s parameters live, at a fixed size no matter how long the context gets.
These aren’t hypothetical. Pairing a single Nvidia H100 with eight Corsair cards in an AFD configuration on GPT-OSS-20B cut time per output token from 2.59 ms to 1.28 ms at batch size 1. In production, Parasail is deploying Corsair alongside its Nvidia Hopper and Blackwell GPUs—GPUs on prefill, Corsair on decode—as part of the Parasail + NVIDIA + d-Matrix collaboration. d-Matrix reports up to 10x faster interactive inference from that pairing and describes it as one of the first commercial-scale examples of heterogeneous disaggregated inference in production.
The next generation goes further. In September 2026, d-Matrix announced a collaboration with Nvidia to integrate its next-generation Raptor accelerators into Nvidia’s MGX rack-scale architecture through NVLink Fusion, letting operators split workloads between Raptor and Vera Rubin. Initial availability is expected in Q4 2027.
There’s an economic upside, too. Moving memory-bound work onto an accelerator gets more value out of the GPUs you already own, freeing them for prefill, training, and other compute-heavy work where their arithmetic throughput actually matters.
What this looks like at rack scale
A disaggregated pipeline only works if accelerators and GPUs can move data between each other fast enough to keep both phases fed. In AFD, for example, the GPU and the accelerator exchange data on every layer of every token.
d-Matrix builds that connectivity on open standards. Corsair is a PCIe Gen5 card that drops into standard servers where a GPU would go, and JetStream, our transparent NIC, carries accelerator-to-accelerator traffic between nodes over standard Ethernet. Rather than routing every transfer through a host CPU, JetStream lets devices write data directly into each other’s memory, keeping the latency penalty per pipeline transfer under 2 µs.
SquadRack brings it all together as a fully qualified rack-scale inference solution, available through partners like Supermicro. Because everything runs on PCIe and IEEE 802.3 Ethernet, operators can mix hardware from different vendors in the same slots and switches without committing to a proprietary fabric.
What to evaluate when adding an AI accelerator to a GPU fleet
Adding a new architecture alongside your GPUs is an infrastructure decision, not just a chip purchase. Ask:
- Is it in production? Look for named deployments and real customer references rather than roadmap slides. Inference at scale surfaces problems that benchmarks don’t.
- Is supply secured? A design only matters if it can be manufactured in volume on a reliable process node. Check the supply chain, not just the tape-out.
- Does it fit your software stack? Framework and orchestration support determine whether adoption is a configuration change or a rewrite. Vendors differ widely in how much of PyTorch and Kubernetes they actually cover.
- What’s the power and cooling footprint? Some accelerators, including Corsair, are air-cooled where comparable GPUs need liquid cooling, which changes what fits in your existing facilities without a retrofit.
- Is it built for heterogeneous deployment? The accelerator should run beside your GPUs on open interconnects, not ask you to swap the fleet out.
How to get started with heterogeneous AI inference
Start with decode-heavy workloads. Interactive assistants, coding tools, and agentic pipelines that chain many short model calls spend most of their time in decode, and they’re where users feel latency the most.
Scope the first deployment to a single workload with a clear latency target. Model size, sequence length, and request patterns all change the result, so a contained workload gives you a cleaner read than a broad rollout. Benchmark on your own models, not just vendor numbers.
Then expand based on what the results show. If offloading decode improves interactive latency and frees up GPU capacity, widen the deployment from there.
Pairing accelerators and GPUs by workload phase is a change you can make without replacing the fleet you have. d-Matrix accelerators take the memory-bound work off your existing Nvidia GPUs and leave them on the compute-bound phases they’re built for. Contact us
FAQs about AI accelerators and GPUs
Do I have to choose between an AI accelerator and a GPU?
No. GPUs excel at compute-intensive work like training and prefill, while memory-optimized accelerators excel at memory-bound decode. Running them together lets each phase of inference use the hardware suited to it. Some accelerators are designed to serve inference standalone, but heterogeneous deployments get more out of the GPUs you already own.
Are AI accelerators compatible with the models I already run?
Usually, as long as the accelerator supports the frameworks you already use. Compatibility lives in the vendor’s software layer, not the model itself, so confirm framework coverage, serving-stack support, and any quantization requirements—converting weights to lower-precision formats—before committing.
Will adding an accelerator improve my existing GPU utilization?
In most cases. Decode is bound by memory bandwidth, so a GPU running it uses little of its arithmetic capacity. Moving decode to a memory-optimized accelerator returns that capacity to prefill and training. The gain is smallest when traffic is already prefill-heavy, such as batch summarization of long documents.
What’s the typical deployment timeline for heterogeneous inference?
It depends on scale and readiness. Rack-scale deployments hinge on hardware availability, power and cooling, and integration with your orchestration stack, so the facility often sets the schedule more than the silicon. Air-cooled, PCIe-based accelerators that fit existing servers shorten that path considerably.
Is heterogeneous inference only for hyperscalers?
No. Neoclouds and enterprises running latency-sensitive generative AI face the same prefill/decode split hyperscalers do, because the split is a property of the workload, not the size of the fleet. PCIe and Ethernet are already standard in most data centers, so the building blocks are already in place.
