The inference metrics that actually matter

Training built the models. Running them is now the bigger job, and the bigger bill. By Deloitte’s estimate, inference has grown from about a third of the world’s AI compute in 2023 to roughly two-thirds in 2026. It has overtaken training as the larger cost.

That leaves every team deploying a model with a question: will it be fast enough for users, and cheap enough to run at scale?

The industry answers both with a single number, usually at the top of every graphics processing unit (GPU) spec sheet: tokens per second. It reads as speed. And since more tokens per second per GPU also means a lower cost per token, it looks like it answers cost as well.

It answers neither. Tokens per second is a lab measurement. It is taken with a single request on an idle machine, or with one big batch run flat out and no latency target at all.

Production looks nothing like that. When hundreds of users hit your service at once, each waiting on their own response, the same GPU delivers well below its spec-sheet figure.

The benchmarks now reflect this. MLPerf, the industry-standard testing platform, throws out any tokens-per-second result that misses its time limit. It counts only throughput a user could actually receive. What it reports is the rate a system sustains while still meeting its deadline.

Three numbers actually predict whether a service is fast and what it costs to run: time to first token, tokens per second, and queue depth.

The clearest way to understand them is to follow a single request through the server.

Follow one request

Someone sends a request to your model. Three things decide how fast or slow it feels.

• Time to first token (TTFT): the wait before the answer begins. Nothing appears until the model has read the prompt and produced its first word. That pause is the time to first token.
  
• Tokens per second: the speed once it starts. After the first word, the rest stream out one at a time. How quickly they come is the tokens per second you keep hearing about.
  
• Queue depth: the wait in line before any of that. If the server is already busy, the request sits in a queue until a slot frees up. How many are ahead of it is the queue depth.

That is the whole life of a request: it waits in line, then it waits for the first word, then it reads the rest. Every slow experience your users have is one of those three going wrong. The skill is knowing which one, because each has a different fix, and fixing the wrong one costs money and changes nothing.

Tokens per second is really three numbers

Of the three, tokens per second is the one on the spec sheet, and it is the easiest to game. It is not one number but three, depending on what you divide by.

• Per user is how fast one person’s answer streams. 
  
• Per server is everyone’s output added together. 
  
• Per GPU is what sets your cost. 

The two numbers that do matter – per user and per server – pull in opposite directions. 

Batch more requests together and the server gets through more work in total, while each person’s answer slows down. 

Think of a bus and a taxi. The bus moves far more people per hour, but taxis move the individual person faster. 

So a tokens-per-second number means nothing until you ask two things about it:

Without both answers, it is a marketing figure.

Different workloads care about different numbers

Which number you fight for depends on the workload, because each kind of workload is slow in a different place. Push on the one that bites, and the other two mostly look after themselves.

A chatbot cares about the wait, not the streaming speed. 

• A person reads the answer as it arrives, and people read at about 238 words a minute, which is roughly 30 tokens a second. 
  
• Once your model streams faster than someone can read, going faster still does nothing they will ever see. What they do notice is the pause before the first word. That is the number to fight for. 
  
• A rough guide from decades of interface research: under a tenth of a second feels instant, a second is fine, ten seconds loses them. MLPerf holds a chatbot to under half a second for the first word.

Search-and-answer, or RAG, also cares about the wait, for a different reason. 

• Here you paste a big chunk of retrieved documents into the prompt, and the model has to read all of it before it can write a single word. That reading is the wait, and a faster chip barely helps. 
  
• What helps is not making the model read the same documents again on every request. Read them once and reuse that work.

Agents care about the wait, over and over. 

• An agent works in steps, and before each step it re-reads everything it has done so far. So it takes in an enormous amount of text and writes very little back, and it does this many times in a row. What the user feels is that first-word wait, stacked up across every step. 
  
• The streaming speed hardly matters. What matters is reusing all that repeated reading between steps. The catch is that agents pause often while they call tools, and a simple cache tends to forget the context during the pause. Get that wrong and an agent that felt quick in testing bogs down once real users arrive.

Reasoning models turn it around. 

• These models think before they answer, and the thinking is just more tokens, a long private draft you pay for and never see. Ask a normal model a question with a two-word answer and it spends about seven tokens. Ask a reasoning model the same thing and it can spend over a thousand getting there. The answer on the screen is a sliver of what you paid for. 
• Here the cost is all in the output, so the number that matters is how fast and cheaply the model can produce a long stream of it.

Voice needs an even rhythm. 

• A voice agent needs the first word fast so the reply does not lag the speaker, and then a steady beat after that. A stutter in the middle of a sentence is worse than a slightly slower reply overall.

The expensive mistake is running one of these on settings meant for another. An agent tuned like a chatbot, or a reasoning model sized as if the visible answer were the whole bill, pays for speed it cannot use while starving the number it actually lives on.

Most slowdowns start in the queue

Queue depth is the metric that most often decides how bad your worst moments get.

When your service suddenly slows down, the model almost certainly did not get slower. Your requests are piling up behind each other. Queues have a nasty habit: the fuller you run the GPU, the sharper the wait climbs. Run it at 70 percent busy and you are already adding waiting time. Past 90 percent, a small rise in traffic causes a huge jump in wait. This is why “just run the GPUs hotter to save money” backfires. The last bit of saving you squeeze out comes straight back as a slow experience for your users.

You can see this directly. A serving engine like vLLM shows you two counts side by side: how many requests it is working on right now, and how many are waiting to start. That second count is the one to watch. If requests are waiting, your problem is the queue, and a faster chip will not fix it. If nothing is waiting but the first word is still slow, your problem is a long prompt taking a while to read.

One edge case is worth calling out. A single very long prompt can hog the GPU while the model reads it, and every short request behind it just sits and waits. One person pasting in a 32,000-word document can slow the first word for everyone else in line. The fix is to chop the long prompt into pieces and let other requests slip in between them, so one big job stops blocking the line.

One caution the other way. An always-empty queue is not the goal either. It means you are paying for machines that sit idle most of the time, waiting for a rush that rarely comes. What you want is a short, steady line that absorbs bursts and still gets people served quickly.

Four things move your cost per token, and only one of them is the chip:

The chip sets the floor. The software running the queue sets the ceiling. Most teams shop hard for the floor and never look up.

Where Neysa Velocis fits

Everything so far points to the same place. The numbers that decide your service are the first-word wait, the streaming speed, the queue behind them, and what all three cost. Most platforms keep them hidden.

The Neysa Velocis platform puts them in front of you, on hardware sized to your workload, with engineers on hand.

• The three numbers, on the endpoint. Time to first token, streaming speed and queue depth sit on one screen next to your spend. You can see which one is hurting before you spend anything trying to fix it.
  
• Single-tenant hardware on bare metal. Run the endpoint on dedicated bare-metal GPUs that are yours alone, so no one else’s traffic lands in your queue and slows your users down. You also choose the precision the model runs at, such as FP8 or INT4, to trade a little accuracy for a lower cost per token where the workload allows it.
  
• Engineers on hand at every step. Neysa engineers help you stand up the serving stack, on a network and storage layer built to keep the GPU fed, behind a defined uptime service level agreement (SLA), ISO 27001 and SOC 2, with data kept in India.

See it on your own traffic

Before you compare a single tokens-per-second number, decide which of the three your workload actually lives on.

Put one live model on a Neysa Velocis endpoint, set your target, and watch what it does.

Talk to a Neysa engineer and size your inference →