vLLM vs SGLang: Why Their Benchmarks Behave So Differently

Have you ever wondered how the same model on the same GPU configuration with the same quantization, yet one framework gives you a TTFT of 200ms and the other gives you 500ms.

Nothing about the hardware or model changed still the same results can be observed.

So what changed?

This blog isn’t about which framework to pick. I already made a decision guide for you HERE.
This one focuses on the aspect most comparison posts skip entirely: 
Why do the numbers actually diverge?

P.S. This blog is fairly technical and assumes basic familiarity with LLMs, transformers, and (ideally) my previous vLLM and SGLang deep dives. If you haven’t read those, some of this may feel like walking into the middle of a movie.

Until now, in the previous two blogs, we’ve mostly talked about memory. We have discussed topics like PagedAttention, KV cache and RadixAttention.

This one is about time and efficiency.

Let’s consider an analogy to start with: Think of the GPU as a single chef in a busy restaurant.

Memory management decides where ingredients are stored and scheduling decides which orders the chef cooks next.

If you schedule poorly, even an infinitely fast GPU will have terrible TTFT.

Before comparing what vLLM and SGLang do differently, we need to understand the two kinds of work they’re constantly trying to schedule: prefill and decode.

Two Very Different Phases of Inference

Every request has two phases.

These are the two complementary steps of serving the same request, but they behave very differently from the GPU’s perspective.

Prefill

Suppose you ask:

“Explain quantum computing in detail. Here’s an excerpt from a research paper…”

Let’s assume that the input prompt is of length 1000 tokens.

Now, the model has never seen these tokens before, so it must process all 1000 tokens.

So it begins processing all at once, in parallel. This is the prefill phase.

As you can guess, the longer input tokens, the longer time is required for the prefill phase.

In this phase, there are massive matrix multiplications happening in the background, and highly parallel GPU utilization occurs. It is an expensive operation, but it happens only once.

Analogy: Reading the entire book before you start writing the summary.

Decode

Now after it has consumed the input context, the model starts generating tokens. Let’s assume the output length is also 1000 tokens for simplicity.

Token 001 ➔ Token 002 ➔ Token 003 … ➔ Token 1000

Each token generation depends on the previous one.

You can’t generate 001, 002, 003 together.

The model has to generate first Token 001, then 002, then 003.

In this phase, one token is generated at a time. As a result, less parallelism can be achieved. This phase is memory-bound and it happens hundreds of times depending on output length.

Analogy: Writing one sentence at a time to write a book. You cannot write Chapter 2 before Chapter 1.

Both phases have huge differences.

Suppose:

Input: 3000 tokens
Output: 100 tokens

Prefill phase handles 3000 tokens in one shot but does huge matrix multiplications for each token.

Decode runs 100 separate iterations but uses prefill-computed KV cache for each token.

They behave completely differently and that’s why modern inference engines schedule them separately, instead of treating a “request” as one uniform blob of work.

Now Imagine 500 Requests Arrive

A, B, C, D… all the way to 500.

Both vLLM and SGLang have the same GPU available to them.

Question: Which requests should they run first?

It’s not that obvious, we will assess different strategies for it.

Strategy 1: First Come First Serve (FCFS)

A ➞ B ➞ C ➞ D

It seems fair at first.

But the problem becomes obvious when A happens to have a 100,000-token prompt.

Everyone has to wait for a long time.

GPU is busy with AAAAAAAAAAAAAAAAAAAAAA
B waiting
C waiting
D waiting

TTFT becomes awful for everyone except A.

Strategy 2: Small Jobs First

Maybe prioritize 10-token prompts before 100,000-token prompts.

Now tiny prompts finish instantly and average latency improves, yayy!

But large prompts might never run.

That’s called starvation, another issue. Imagine a 100,000-token request sitting in a queue forever while 100 tiny requests arrive every second. The scheduler keeps saying “small first, small first, small first” forever. The large request never gets its turn.

Schedulers have to prevent this. Every request should eventually make progress, not necessarily fast, but eventually.

So schedulers are constantly balancing fairness, latency, and throughput, and improving one usually means making a tradeoff somewhere else.

There’s no free lunch here, and this is exactly the kind of tradeoff that makes vLLM’s benchmark numbers look completely different from SGLang’s, even on identical hardware.

Let’s understand another problem vLLM and SGLang face and how do they deal with it.

Continuous Batching, Revisited

We covered this in the vLLM blog, but here’s the part that matters for this investigation.

Old batching:

Batch: A, B, C ➞ A Finishes but has to wait for B and C to finish too
New batch: D, E keeps waiting

GPU waits and there’s lots of idle time.

So the solution continuous batching brings is:

A finishes ➞ Immediately insert D ➞ Batch becomes B, C, D

Then C finishes, insert E. Batch becomes B, D, E.

The batch is constantly changing. The GPU almost never goes idle.

But here’s the hidden problem nobody mentions in the docs:

Suppose 100 decode requests are already running and a new user arrives. So their request needs prefill now, but as we know prefill is expensive.

So what do we do? Should we stop decoding to run it?

If we run prefill immediately, the new user gets fast TTFT, but the existing users’ generation slows down; if we wait for the current decoding work to finish, the existing users stay happy while the new user might wait seconds before getting their first token.

This is the core scheduling problem, and it’s exactly where inference engines like vLLM and SGLang start to diverge in behavior.

Chunked Prefill

The solution many engines reach for is chunked prefill.

Instead of running the entire 100,000-token prompt in one go:

512 tokens
↓
decode step for others
↓
512 tokens
↓
decode step for others
↓
512 tokens

The long prompt no longer monopolizes the GPU.

Think of downloading a huge file. Instead of downloading 20GB and blocking everything else, your OS downloads it in chunks while still serving other apps.

Exact same philosophy is being used here, just for tokens instead of bytes.

Tradeoff: Too much chunking adds scheduling overhead and lowers throughput. Too little chunking brings back terrible TTFT. Chunk size is a balancing act, not a solved problem.

Another thing that matters in inference engines is how do you admit new requests.

Smarter Admission Control

Imagine the GPU is already decoding 300 users and a new request arrives.

Should we admit it now?

A naive scheduler could admit everything immediately, but then the GPU gets overloaded and everyone slows down.

The alternative is to queue new requests, but that increases TTFT.

Modern schedulers instead estimate current KV cache usage, current batch size, decode cost, prompt length, GPU occupancy and memory remaining and then decide: admit now, wait, chunk, or delay.

Analogy: Think of airport security, if 50 people already entered, security temporarily pauses entry, not because new passengers are unwanted, but because letting everyone in at once slows down everyone already inside.

Why TTFT Actually Grows

TTFT isn’t just the time spent doing prefill. It includes:
Queueing delay + Prefill time + First decode step

The model’s raw compute speed hasn’t changed. What changes is how long a request waits before its prefill even begins.

Imagine GPU busy with Request A having 1000 token input again, and a new request comes, it’ll keep waiting as we’ve deduced a million times already.

If the scheduler admits the new request’s prefill quickly, TTFT stays low. If it waits through several decode iterations before admitting it, TTFT grows even though nothing about the model got slower.

This is exactly why two systems with identical GPUs and identical models can post very different TTFTs: the difference isn’t necessarily compute, but how they decide who gets GPU time next, and when.

Where vLLM and SGLang Actually Diverge

vLLM popularized continuous batching and pairs it with PagedAttention to maximize throughput and admit new requests efficiently.

SGLang builds on the same foundation but invests heavily in scheduling, prefix reuse, RadixAttention, chunked prefill, and admission policies.

Those choices can reduce unnecessary computation and queueing delay for workloads with significant shared context, while still maintaining high throughput.

Once you understand scheduling this way, the benchmark results you’ve probably seen — where SGLang reports lower TTFT without sacrificing throughput — stop looking mysterious.

They’re consequences of scheduling strategy and workload characteristics, not simply faster matrix multiplication.

Up to this point, scheduling explains how requests compete for GPU time.

But scheduling alone still doesn’t explain why two seemingly similar benchmarks can report completely different winners.

To understand that, we need to look at one more variable that rarely appears in benchmark charts.

The Hidden Variable That Changes Everything

This is where benchmark comparisons get particularly interesting.

You might find one chart saying SGLang is 35% faster, another showing they’re basically identical, and another where vLLM wins.

The reason is that they’re often testing different workloads, specifically different amounts of shared context between requests.

This is called the cache hit rate, and it can be one of the biggest factors in this comparison.

Let’s consider two examples:

First example has Cache Hit Rate → High

Imagine 100 requests, all starting with:

System: You are a helpful assistant.

Only the user’s question changes each time.

Every request shares the system prompt, therefore hit rate is extremely high.

Or take use cases like RAG where every user is asking questions about the same Tesla 2026 Annual Report.

Context: <same 5000 tokens> + Different question each time

The prefix is huge and shared and SGLang can reuse almost all of it due to its RadixAttention mechanism and barely anything needs recomputing.

Now consider Example 2 where Cache Hit Rate is Low:

Now imagine a translation service where every user uploads a different document.

Document A
Document B
Document C

Nothing overlaps and hit rate drops to almost zero. Prefix caching has nothing to reuse, RadixAttention’s tree ends up with almost no shared branches to walk.

Think of it like YouTube’s caching. If everyone’s watching the same viral video, caching is magic. If everyone’s watching something different, the cache barely helps anyone.

Where This Actually Bites

Long prompts make this dramatic.

An 8000-token prompt where 7900 tokens is system prompt and is shared across requests only needs 100 tokens of fresh computation. That’s a massive saving and exactly why coding copilots, RAG systems, and agent loops can benefit heavily from prefix reuse.

Agent loops are the extreme version of this. Every reasoning step re-sends almost the entire conversation history plus one new tool output:

Loop 1: Conversation + New Tool Output

Loop 2: Conversation + Previous Tool Output + New Tool Output

Almost everything overlaps turn to turn.

This is exactly the kind of workload where prefix reuse can provide a significant advantage.

When vLLM Is Just As Good

This is the part people miss.

Consider use cases like website chatbots, where your workload is independent, and users send random, unrelated prompts, prefix caching barely matters in these cases.

Performance then comes down to scheduler efficiency, kernel speed, and memory management, and in that world, vLLM is often just as good, sometimes within a few percent.

So two fair benchmarks, run on genuinely different workloads, can honestly report opposite winners because they’re measuring different points on the same cache-hit-rate spectrum.

Two Different Architectures, Not Just Two Different Optimizations

There’s a deeper difference sitting underneath all of this, and it’s architectural, not just a feature checklist.

Features like structured generation and tool calling are typically layered on top of these serving engines as additional decoding constraints.

Neither approach is wrong. They’re answers to two different questions:

vLLM asks:

“How do I serve as many LLM requests as possible with excellent GPU efficiency?”

SGLang asks:

“How do I efficiently execute LLM-powered programs that involve repeated context, structured outputs, tool calls, and agent loops?”

That’s why vLLM can lead ahead on raw throughput in serving benchmarks, while SGLang can pull ahead specifically in agent-heavy workloads.

And that’s also why neither framework actually made the transformer itself any faster.

They just organized the work around it differently.

Conclusion

The important thing here isn’t that one framework is simply faster than the other.

They’re organizing the same GPU’s time differently, and your use case and workload determines which approach has more room to help.

Neither framework made the transformer compute a single matrix multiplication faster than the other.

What changed is how much unnecessary work each one avoids, and how intelligently each one decides who gets the GPU next.

That’s the actual answer to “Why do these benchmarks disagree?”

It’s not necessarily a bad benchmark or misleading marketing. Often, they’re simply answering different questions with different workloads.

If you enjoyed this one, the decision-guide version on which framework you should actually pick is HERE.