Llama 3 vs Qwen 3: Benchmarking in Production

Introduction

Qwen 3 or Llama 3? Does FP8 actually move the needle or is it just a memory trick? Does the MoE architecture on the 235B hold up when concurrency climbs? 

We keep getting asked the same set of questions. So, we ran the tests to find out some of the answers. 

Four models. Two architecture families. Real production conditions – meaning concurrency levels that actually stress the system, not single-user cherry-picked numbers. And here’s what the data showed: 

What We Ran 

Four models across two architecture families, all on NVIDIA H100 SXM GPUs with 80 GiB memory per GPU: 

• Llama-3.1-8B-Instruct (128k context, dense) 
• Llama-3.3-70B-Instruct (128k context, dense) 
• Qwen/Qwen 3-32B (32k context, dense, 32.8B parameters) 
• Qwen/Qwen 3-235B-A22B (32k context, MoE, 235B total parameters, ~22B active per token) 

Each configuration ran on 1,000 prompts at 1,000 input tokens and 100 output tokens. Concurrency tested at 10, 50, and 100 simultaneous users. GPU count ranged from 1 to 8 depending on what the model could actually support. Both FP16 and FP8 were tested. 

Three metrics across every run: 

• Throughput (tokens per second) 
• Time to first token (TTFT, in milliseconds) 
• Median end-to-end latency (in milliseconds)

What the Data Showed 

Here’s a quick glimpse of the results before we get into each finding: 

• FP8 on the same GPU count nearly doubles throughput for Llama-3.1-8B-Instruct – but the gains are more complicated for Qwen 3-32B, where FP16 actually outperforms FP8 at moderate concurrency 
• Adding GPUs past the right threshold reduces throughput rather than improving it, at least for the smaller model 
• Qwen 3-32B posts throughput at high concurrency that competes closely with Llama-3.3-70B-Instruct, but trails significantly on TTFT 
• Qwen 3-235B-A22B hits a latency wall at concurrency 50 that rules it out as a general-purpose endpoint 

FP8: Not Just a Memory Trick 

FP8 only matters when it improves real workload behavior, which is why teams need to evaluate it inside AI infrastructure as a service⁠ rather than as an isolated precision setting.

Llama-3.1-8B-Instruct is where the gain is most dramatic: 

• Memory drops from ~15 GiB per GPU on FP16 to ~8.5 GiB on FP8
• At concurrency 100 on 4 GPUs, FP16 delivers ~5,352 tokens per second. FP8 on the same hardware delivers ~10,724, which is nearly 2x. 
• TTFT follows the same pattern: at concurrency 10 on 8 GPUs, FP8 cut it from 48ms to 15ms 

Qwen 3-32B is more complicated, and worth paying attention to: 

• FP16 memory exceeds 61 GiB per GPU, forcing multi-GPU setups from day one. FP8 brings that down to ~32 GiB, making single-GPU configurations viable for the first time 
• At concurrency 50 on 8 GPUs, FP16 actually outperformed FP8 – 3,057 versus 2,198 tokens per second on throughput, 1,479ms versus 2,179ms on latency. FP8 was slower, and not by a small margin 
• At concurrency 100, throughput converges (~6,049 FP16 versus ~6,099 FP8), but FP8 cuts TTFT from 327ms to 176ms 

The takeaway for Qwen 3-32B: FP8 pays off clearly at high concurrency. At moderate load, FP16 is the better choice. 

Benchmark AI Models in the Real World

Test model performance on production-ready AI infrastructure.

Explore Velocis

Concurrency Is Where Things Get Interesting 

At higher concurrency, high throughput in inference⁠ becomes the practical measure of whether a deployment can hold up under production traffic

Single-user latency looks fine for almost every model. The question is what happens when 50 or 100 users hit the system simultaneously. 

Llama-3.1-8B-Instruct handled concurrency well throughout: 

• At concurrency 100 on 4 GPUs: sub-800ms end-to-end latency, throughput scaled cleanly
• Most predictable configuration in the study 
• Llama-3.3-70B-Instruct showed bigger variance depending on configuration: 
• 4 GPUs FP16, concurrency 50: 5.6 seconds end-to-end latency 
• 8 GPUs FP8, concurrency 50: 1.47 seconds – same model, same concurrency
• 8 GPUs FP8, concurrency 100: 1.76 seconds end-to-end, TTFT of 84ms 

Qwen 3-32B held its own on throughput but gave ground on responsiveness: 

• At concurrency 100 on 8 GPUs FP8: ~6,099 tokens per second – closely matching Llama-3.3-70B-Instruct (~5,059 tokens per second) on the same configuration
• But TTFT tells a different story: 176ms for Qwen 3-32B versus 84ms for Llama-3.3-70B-Instruct 
• Qwen 3-235B-A22B hit a ceiling fast: 
• Concurrency 10: 2.6 seconds end-to-end (workable for slow analytical tasks)
• Concurrency 50: 5.7 seconds 
• Concurrency 100: 9.1 seconds 

At nine seconds, it’s not an inference endpoint. It’s a queuing problem. 

More GPUs Don’t Always Mean More Throughput

This is one of the clearest examples of the GenAI product trilemma⁠, where more infrastructure does not automatically solve speed, cost, and control together.

For Llama-3.1-8B-Instruct in FP16 at concurrency 50: 

• 2 GPUs: ~3,188 tokens per second 
• 4 GPUs: ~7,773 tokens per second (the sweet spot at this load level)
• 8 GPUs: ~5,224 tokens per second (throughput dropped) 

Adding GPUs past four actively hurt performance at concurrency 50. The 8-GPU configuration only justifies itself at concurrency 100, where it delivers 11,538 tokens per second. 

Qwen 3-32B scaled differently – throughput gains from additional GPUs remained meaningful across concurrency levels. The 8-GPU deployment genuinely helped. 

Qwen 3-235B-A22B has no scaling decision to make. 8 GPUs is the minimum. That changes the economics of the deployment before you’ve even looked at a workload. 

What to Deploy, and When 

Once teams compare model behavior under real concurrency, the decision starts to look less like model selection and more like an AI neocloud⁠ infrastructure decision.

Llama-3.1-8B-Instruct, FP8, 4 GPUs – the default for most production systems: 

• RAG pipelines, conversational AI, enterprise search, summarization
• ~10,724 tokens per second at concurrency 100, sub-800ms end-to-end latency
• At concurrency 50, 4 GPUs outperforms 8 GPUs – don’t over-provision 
• Qwen 3-32B, FP8, 8 GPUs – strong for throughput-heavy analytical work within a 32k context window: 
• Handles concurrent load well, memory footprint stays manageable
• TTFT under high concurrency is higher than Llama-3.3-70B-Instruct – if first-token speed matters, the 70B is the better call even though throughput is comparable
• FP8 advantage is most pronounced at concurrency 100; at concurrency 50, FP16 actually wins on throughput and latency 

Llama-3.3-70B-Instruct, FP8, 8 GPUs – right for long-context, high-stakes workflows: 

• Compliance analysis, KYC, document synthesis, executive decision support
• 84ms TTFT at concurrency 100 is strong for a model this size 
• 128k context window opens up document-heavy workloads that Qwen 3-32B can’t support 

Qwen 3-235B-A22B, FP8, 8 GPUs – specialist tool, not a general-purpose endpoint: 

• Sequential document review and low-concurrency analytical pipelines where output quality is the priority 
• Latency crosses 5.7 seconds at concurrency 50 and 9 seconds at concurrency 100
• Know your concurrency ceiling before committing to this deployment 

Why This Matters for Inference Infrastructure 

Production inference is a multi-variable problem: 

• Context windows affect memory allocation 
• Quantization changes what configurations are even viable 
• Concurrency reshapes latency in non-linear ways 
• GPU count interacts with concurrency to produce results that sometimes go backwards 

This is why Neysa approaches inference infrastructure as a continuously evolving production environment. The benchmark data surfaced through Velocis – concurrency ceilings, TTFT curves, FP8 crossover points, and GPU scaling behavior give teams the visibility to provision around actual workload demands rather than synthetic test assumptions. The gap between a deployment that holds up under load and one that degrades usually comes down to decisions made before traffic arrives. 

Conclusion

A few things the data made clear: 

• FP8 on 4 GPUs nearly doubles throughput over FP16 on the same hardware for Llama-3.1-8B-Instruct
• Adding GPUs past the optimal count can reduce throughput at moderate concurrency
• Qwen 3-32B matches Llama-3.3-70B-Instruct on throughput at high concurrency but trails on TTFT – which matters for streaming interfaces
• Qwen 3-235B-A22B is a specialist model that hits a latency wall well before enterprise-scale concurrency  
• For Qwen 3-32B, the FP8 advantage is concurrency-dependent: better at 100, not always better at 50

FAQ