If vLLM already solved LLM serving, why did SGLang appear?
Introduction
When Kimi K3 shipped its open weights in July 2026, the coverage focused on what it scored.
A 2.8 trillion parameter model, the largest open-weight release in history, topping several major benchmarks within days of release. But what got significantly less attention was the hardware floor. A minimum of eight H100 80GB GPUs just to load the model. Not to run it well, or to serve production traffic – just to have it in memory.
That gap between the headline and the hardware requirement is not a footnote. It points to a fundamental question, that every enterprise putting serious AI investment to work has to reckon with – when models are built on a fundamentally different architecture than the transformer models of a few years ago, do the infrastructure assumptions carry over? The answer, in most cases, is no.
Not All Parameters Are Created Equal
The model architecture behind Kimi K3, and behind a growing list of the most capable models available today, is called Mixture of Experts (MoE). To understand why the infrastructure picture looks different, you have to understand what that actually means.
A standard dense transformer model activates every parameter for every token it processes. The entire model is engaged, every forward pass, regardless of what the input is. This is computationally reliable, but wasteful in the sense that most of those parameters are probably not highly relevant to the specific token being processed at any given moment.
Mixture of Experts models take a different approach. Instead of one large feed-forward network per transformer layer, they have many smaller ones, each called an expert. A lightweight router is called a gating network. It examines each token and routes it to the small subset of experts most likely to handle it well. The rest of the experts do nothing for that token. They sit dormant, waiting for inputs that match their specialization.
Kimi K3 has 896 experts and activates 16 per token. Mixtral 8x7B has 8 experts and activates 2.
In both cases, the model’s total parameter count is enormous, but the compute actually required per token is a fraction of what a comparable dense model would need. Mixtral carries 47 billion total parameters but runs at roughly the compute cost of a 12 billion dense model. Switch Transformers, one of the early landmark MoE architectures from Google, demonstrated a 4x pre-training speed improvement over a dense model of equivalent size.
The headline is simple: with Mixture of Experts, you pay for what you activate, not for everything you have. That is the efficiency case. The infrastructure case is far more complicated.
Where Do The Other Experts Live?
Here is the part the benchmark writeups tend to skip. The experts that are not activated for a given token still have to exist somewhere. During training, gradient computation requires access to the full parameter set. Which means all experts must be in memory at all times. During inference, all expert weights have to be loaded even if most of them will not fire for any given request.
A standard MoE configuration with 64 experts, each consisting of two feed-forward layers at typical dimensions, carries roughly 134 million parameters per expert. Multiply by 64 experts and you have 8.6 billion parameters in the expert layers alone, before attention mechanisms, embeddings, and the optimizer state that doubles or triples the memory requirement during training. No single H100 with 80GB of memory holds this alongside everything else the training process needs.
Expert parallelism is the architectural answer to this problem.
Rather than asking any single GPU to hold all experts, you distribute the expert pool across the cluster. With 64 experts and 8 GPUs, each device holds 8 experts. The gating router is small enough to replicate on every device. When a token needs an expert that lives on a different GPU, it travels there over the interconnect. The expert processes it, and the result returns. That round-trip, called dispatch and gather, is the defining operation of expert parallelism, and it happens at every MoE layer, on every forward pass, for every token in the batch.
The Two Things That Are Hard to Get Right
Understanding expert parallelism at the concept level is one thing. Running it in a production system without surprises is another, and there are two places where teams consistently underestimate the complexity.
a) Load Balancing: Without intervention, a gating router quickly settles on a few favorite experts and keeps sending them most of the traffic. The GPU hosting a popular expert becomes the bottleneck for the entire cluster while others sit underutilized. Addressing this requires an auxiliary loss that nudges the router toward even distribution. A capacity factor that caps tokens per expert per batch, and a clear policy for what happens when an expert hits that cap. Get it wrong and you are burning expensive GPU time on a configuration problem.
b) Parallelism Stacking: The second is that expert parallelism rarely runs alone. The number of experts in a model sets a hard ceiling on how far EP scales, so most production deployments combine it with tensor and pipeline parallelism. NVIDIA’s research on a 1.8T MoE model found that the right combination delivered 3x higher GPU throughput over a tensor-only setup with no interactivity cost. The right configuration is different for every workload, and finding it requires deliberately working through the options .
What This Means When Enterprises Start Evaluating These Models
The MoE architecture is not a niche research direction at this point. Kimi K3 is the largest open-weight model in the world and it is MoE. DeepSeek’s V2, V3, and R1 models are MoE. The models that enterprise teams are now piloting for internal AI applications, coding assistants, document workflows, and customer-facing inference are increasingly built on this architecture. Therefore the infrastructure question that comes with them is not hypothetical.
Several things become true when you are running MoE models in a real environment.
The GPU count is necessary but not sufficient: the interconnect fabric between those GPUs determines whether the dispatch-and-gather operations that expert parallelism depends on complete fast enough to be invisible to the rest of the workload.
NVLink handles bandwidth within a node; InfiniBand handles traffic across nodes. When either of those is undersized, the all-to-all communication operations that run at every MoE layer become the rate-limiting step, and adding more GPUs does not help.
The parallelism configuration also requires deliberate setup: the same model running EP8DP8 versus EP16DP4 on identical hardware will deliver meaningfully different throughput and latency profiles, and neither of those configurations is what you want if your workload justifies adding pipeline parallelism to the mix.
None of this is insurmountable, but it is specific in ways that generic cloud GPU access tends not to accommodate well. Infrastructure built for MoE-scale distributed training and inference has the interconnect topology, the node configuration, and the multi-GPU memory architecture that make expert parallelism run the way the benchmarks suggest it should.
That is what Neysa Velocis is designed to offer – GPU infrastructure purpose-built for the class of workloads that these models represent, rather than adapted from infrastructure designed for something simpler. Understanding the architecture is the starting point. Getting the infrastructure right is what makes the benchmark translate to something useful.
