In the early days of transformers-based AI workloads, generation speed wasn’t that big of an issue. Users generally tolerated slower inference due to novelty, and serious use cases outside of…Read More
Time-to-first-token, or TTFT, has generally been one of many key performance indicators for AI-based applications. Consumer-facing chatbots, for example, might need it to be very low, while enterprise cases can…Read More
d-Matrix’s advantage over classic GPU architecture is extremely low latency, ultra-efficient small batch inference. We approach each of those problems with a large range of techniques and must excel at…Read More
In today’s rapidly evolving AI landscape, it’s clear that inference — not just training — is becoming the new scaling challenge. As models grow in size and capability, the infrastructure…Read More
We are inevitably moving toward a world where AI applications need substantially better performance, and classic GPUs alone can’t keep up. And at the same time, customers need flexibility in…Read More
Large-scale AI workloads are deployed by stringing together many cards and many nodes. Any increase in model size, context length or users impacts the number of cards needed. Gone are…Read More
AI adoption is accelerating at an unprecedented pace, but the economics of running AI models are becoming unsustainable. And the cracks are beginning to show. As more people rush to…Read More
We’ve become accustomed to a brand new model coming out every few weeks that one-ups the last one at this point. Developers don’t sit with just a single model—in fact,…Read More
The future of AI workflows is almost certainly going to be multi-modal agents. But rather than incredibly complex, compute-hungry multimodal models, there’s already a much easier pathway to get there. …Read More
‘AI Inference’ has been trending everywhere, from keynote speeches to quarterly earnings reports and in the news. You have probably heard phrases like “inference-time compute”, “reasoning” and “AI deployment” and…Read More
