Disaggregated AI pipelines have offered an immediate path toward more efficient, extremely low-latency interactive experiences.
This includes implementing techniques like prefill/decode disaggregation, speculative decoding, and attention-FFN disaggregation. We’ve already proven out speculative decoding in action using Corsair with GPUs.
We’ve now leveraged the extreme low-latency performance of Corsair by implementing attention-FFN disaggregation (AFD) by combining the advantages of an Nvidia GPU managing attention, embeddings, and expert routing while Corsair manages expert computation with its ultra-high bandwidth SRAM-based memory architecture.
This is possible because of Corsair’s highly scalable SRAM memory architecture which gracefully handles fixed-memory tasks like running the active experts in each inference. All experts—consisting of 19.12B total parameters for GPT-OSS-20B—live in SRAM, with 4 experts active (3.6B parameters) per inference.
We can generate 845 tokens per second per user in this heterogeneous configuration with the efficiency advantages Corsair brings to AI inference systems. With 64 concurrent users, this heterogenous AFD setup delivers a total throughput of 20,415 tokens per second.
Let’s break down how this all works.
Making attention-FFN disaggregation work
GPT-OSS-20B is a lightweight mixture-of-experts (MoE) model that made sense as a natural starting point for implementing attention-FFN disaggregation. The experts make up roughly 19.12B out of a total 20.9B parameters per its model card. These experts at 4-bit precision fit on an eight-card Corsair server with 16GB of total SRAM.
Corsair is then paired with one 80GB H100 that manages the attention layer and KV cache including embeddings and top-4 expert routing. GPT-OSS-20B activates 4 experts out of 32 total amounting to roughly 3.6B total active parameters. For a batch size of 64, two H100 GPUs share eight Corsair accelerators.
Corsair operates using SRAM, which enables memory bandwidth speeds that are 45x (150 TB/s) of what’s available with standard HBM3 on H100 (and roughly 9x the bandwidth of the 18TB/s that HBM4 offers in newer GPUs). The set of experts runs at extremely low latencies compared to GPU-only configurations by sitting directly adjacent to the compute. Rather than a large series of trips to and from HBM for retrieving the weights each time, AFD requires two: shipping the hidden state from the attention layer processed on the GPU to the Corsair cluster and then returning the results from the FFN to the GPUs.
The compute intensity here makes it a poor fit for GPU-only pipelines, and why implementing heterogeneous pipelines with Corsair offers a massive performance improvement. GPUs are extremely inefficient for small batch sizes—all the way down to a batch size of one—that are necessary to preserve a highly interactive, snappy experience. At time of inference, H100s must stream the selected expert weights (four per inference) from HBM, which can transmit roughly 3.35 TB/s of data.
Implementing Corsair and JetStream to massively accelerate AI inference
An Nvidia H100 is paired with eight Corsair cards using JetStream, our latency-optimized transparent NIC. JetStream enables Ethernet-based communication with a network bandwidth of 400 Gb/s per cluster. JetStream also enables device-initiated communication, where the GPU kernel writes data directly into Corsair memory and the Corsair kernels write their results directly back into GPU memory with no host involvement.
As a result, neither side is waiting on a host CPU. The H100 operates with a single CUDA graph while each Corsair—hosting four experts for every layer—starts running as soon as the data arrives. All experts for all layers are run as a single Corsair graph as well, with no graph-breaks in between layers. (You can see more about the technical architecture of JetStream here.
GPT-OSS-20B includes 24 total layers, resulting in 24 round trips between Corsair and the Nvidia H100, plus one additional one for the LM head. JetStream enables this through an Ethernet fabric built upon open standards, allowing Corsair to be a pure plug-and-play for any PCIe compatible rack architecture.
The GPU generates the 2,880-dimension GPT-OSS-20B hidden state vector during the attention process as normal and selects the token’s four target experts. Here, the H100 would normally fetch the expert weights from memory and run the feed-forward network. Instead, the GPU is sending that state and the target expert assignments to Corsair.
That round-trip of writing the hidden state to card memory, running the experts, and writing the results back, incurs additional latency. Corsair, however, processes the feed-forward network while the GPU is pre-fetching layer weights for the next layer. The GPU writes the hidden state to two “root” Corsair cards, which then relay data to the complete Corsair cluster over PCIe. With the overlap, the actual pipeline only “perceives” a minimal exposed latency of the total round trip as waiting.
Managing the LM head
Each Corsair card holds one shard of the LM head that scores every word in the model’s vocabulary to select the final result. Corsair has two memory tiers: SRAM and DRAM. We take advantage of both forms of memory available to manage the LM head while ensuring an ultra-low latency experience.
The LM head is streamed in from DRAM as Corsair completes the remaining layers. Each Corsair then processes the LM head using its vocab shard, sending its proposed result back to the GPU. The GPU selects the best of the eight records, and Corsair subs the swapped-out layers back into SRAM for the next token.
As a result, we cut down the LM head’s time by 10x, while the off-chip memory traffic is essentially hidden.
Massively accelerating GPU-only configurations
On its own, one H100 running GPT-OSS-20B at a 1K context length delivers a TPOT of 2.59ms. Implementing Corsair into a heterogeneous AFD pipeline brings the TPOT down to 1.28ms (a roughly 50% reduction!).
In this heterogeneous AFD configuration, the work Corsair’s has to do for each inference remains predictable and dependent on batch size because the FFN represents a fixed-memory object rather than the context length. The GPU’s share meanwhile rises as it manages growing context for a user. That gives us an ideal split for AFD pipelines: GPUs deal with components that are growing, Corsairs deals with components that are fixed.
Corsair plug-and-play air-cooled infrastructure, combined with JetStream, enables high-performance AI inference in standard data center configurations that scales in a more sustainable manner.
Scaling AFD with larger batch sizes
The batch size of one delivered shows the latency floor. GPUs rely on larger batch sizes to maintain utilization, but that comes at the expense of interactivity.
At a batch size of 64, each of the 64 tokens in production are selecting four experts. In most scenarios here, you likely have all 32 experts doing work at a given time, and the GPU streams every expert’s weights on every step. You generate more throughput but are watching the TPOT quickly rise.
The experts, however, never leave Corsair. With batch size of 64, we have two H100 GPUs share the same eight Corsair cards, with each GPU running 32 user requests. The traffic is staggered, enabling a full 64-user decode step in 2.46ms at 129 context length. This configuration delivers a total throughput of 20,413 tokens per second across all users with a 1K/1K context length configuration. (We’ll discuss how this performs side-by-side GPU-only configurations at BS64 to show how Corsair substantially improves interactivity in a later blog.)
Disaggregation beyond Corsair
Corsair can now support AFD setups to massively accelerate AI inference while scaling in a sustainable, graceful way. Corsair can also scale up and scale out to support models in the hundreds of billions of parameters in both speculative decoding and AFD configurations, targeting models as large as GLM 5.x and Kimi K3.
AFD represents a deterministic approach compared to speculative decoding, where draft models are generating results to be verified by the GPU. But both can work in tandem to massive accelerate AI inference workloads.
Model sizes are still exploding at the same time customer expectations are rising for agent-enabled AI inference products. Throwing more GPUs at the problem to sustain those premium low-latency experience is one way to keep up at the cost of letting the majority of the compute power unused.
Or, you can implement a disaggregated solution using memory-optimized accelerators through Corsair’s air-cooled, PCIe-based SRAM architecture.
The future is disaggregated—and it’s here today. If you’re interested, we’d love to work with you.
