Conversations around fast inference typically focus on one approach: blazing fast token generation with gargantuan models. Both still play an important part in many circles. Models come in huge flavors, including Qwen (235B)… Read More
The first cloud computing revolution triggered a tsunami-grade buildout for data centers—both for emerging hyperscalers like Amazon Web Services, and companies re-orienting their systems around cloud-based operations. At the time, we were… Read More
d-Matrix’s advantage over classic GPU architecture is extremely low latency, ultra-efficient small batch inference. We approach each of those problems with a large range of techniques and must excel at… Read More
In today’s rapidly evolving AI landscape, it’s clear that inference — not just training — is becoming the new scaling challenge. As models grow in size and capability, the infrastructure… Read More
We are inevitably moving toward a world where AI applications need substantially better performance, and classic GPUs alone can’t keep up. And at the same time, customers need flexibility in… Read More
Large-scale AI workloads are deployed by stringing together many cards and many nodes. Any increase in model size, context length or users impacts the number of cards needed. Gone are… Read More
