The Economics of Running AI Inference on AMD MI350X

Every time someone uses an AI model, it runs inference, reading the request and producing an answer, token by token, and you pay for every token. It runs every day the model is live, and grows with every user.

There are two forces pulling against each other, when it comes to tokens – 

• Tokens keep getting cheaper, a million of them fell from about $20 to $0.40 in three years. 
  
• But usage is climbing far faster: reasoning models spend hundreds of tokens thinking before they answer, agents loop through thousands per task, and token volume across the industry has risen roughly 68-fold in a year. 

So the bill keeps rising even as each token gets cheaper, and in ways that are hard to predict. Making inference the biggest line in most AI budgets, and one that’s hard to forecast. It’s the same three-way pull between throughput, latency and cost every production AI team ends up managing.

Enter MI350X.

The MI350X serves each token for less: its 288GB memory runs a large model on fewer GPUs [v/s other GPUs with lower memory], and its FP4 engine pushes more tokens through each one, at roughly $0.17 per million. Neysa Velocis runs that chip as a production service, dedicated endpoints your app calls, with the storage, networking and orchestration behind them, and spend you can trace. Use them together, and you get reliable AI inference at predictable speed and cost, even as you scale.

At this point you might be wondering: “will my AI model even run on AMD chips?” – It does, with no rewrite, and we cover it in the companion post, “Will Your AI run on AMD?”. 

Where the savings come from

Start with the chip. Two things make the MI350X cost effective  per token, and both come down to serving more workloads on less hardware. 

The first lever is memory. 

A model has to fit in a GPU’s memory to run, and the MI350X carries 288GB of it on a single card, enough to hold a large model that would otherwise be split across two or three GPUs. Fewer GPUs for the same model

The second lever is precision. 

An AI model consists of billions of variables. Storing each one with fewer digits shrinks the model’s overall size, allowing it to run much faster. 

The MI350X handles the newer FP4 and FP6 formats natively, so it packs each model tighter and pushes more tokens through the chip for the same electricity, at a loss of accuracy most workloads never notice. Together, memory and precision land the cost that independent testing has measured: DeepSeek R1 at about $0.17 per million tokens, GLM-5 near $0.22, and on Llama-class models, up to 40% more tokens per dollar than a competing chip.

Fast, and steady as you scale

Cheap only helps if it’s also fast, and the MI350 series is fast. MLPerf is the industry’s go-to benchmark, and in its 2026 round, a single eight-GPU server served gpt-oss-120B at about 95,000 tokens a second and Llama 2 70B over 100,000, and scaled past a million a second across eleven servers.

Raw throughput is only half of what matters, though. Total tokens per second tells you how much the whole server produces; it says nothing about how any one user’s request feels. 

What a user feels is responsiveness, how quickly the first word appears, and how many tokens a second they get while thousands of other requests are in flight at the same time. The MI350 series holds those steady as that concurrency climbs, so a spike in traffic doesn’t turn into a spike in latency. That’s the predictable-speed half of the promise: the service stays fast as it grows, not just on a quiet benchmark.

Inference needs more than a chip

A fast, cheap chip still isn’t inference. Turning it into a service real users can hit takes a whole stack around it: 

• Fast storage so the GPU never sits waiting for the next batch of data
  
• Networking quick enough to link many GPUs without stalling them
  
• Orchestration to place each request and recover when a machine fails
  
• Secure endpoints for requests to arrive at
  
• Monitoring that shows speed, cost and errors as they happen. 

Miss any one of these and even the fastest chip sits idle much of the time. This is the gap between owning GPUs and running inference in production.

Neysa Velocis runs AMD MI350X in production

Neysa Velocis covers the whole stack – storage, networking, orchestration, secure endpoints and monitoring built around the chip, with the MI350X as one of the GPUs you can run on it, and the only Indian cloud recommended in SemiAnalysis’s ClusterMAX rating of GPU clouds, an independent grade of exactly the things a chip can’t do alone. 

Every part of it is built to keep inference reliable and its cost in view:

• Dedicated, single-tenant endpoints. Your model runs on hardware that’s yours alone, no noisy neighbors competing for the GPU, so latency stays steady. You pick the quantization (FP8, INT4, AWQ) to trade a little accuracy for lower cost per token where you can [note: Bare metal only.
  
• Cost per token, based on the infrastructure selected, in front of you. Per-minute, per-project metering sits next to the metrics that decide inference, time to first token, queue depth, so cost and performance are on the same screen.
  
• Every form factor, plus the platform around it. GPU VMs and containers, self-serve BMaaS, dedicated bare metal, and VKE for Kubernetes, with managed notebooks, ready-to-run inference endpoints, and VKE on top. Prototype in a notebook, save your model, then self-serve it on a GPU VM (e.g. with vLLM) or scale out on VKE.
  
• Infrastructure built for AI, and proven. WEKA storage and a 3,200 Gbps Arista network with RoCE v2 and GPU Direct keep the GPUs fed at full speed, behind a defined uptime SLA, SOC 2, ISO 27001, 27017 and 27018, CSA STAR, and India data residency. Velocis is the only Indian cloud recommended in SemiAnalysis’s independent ClusterMAX rating.
  
•  Engineers who run AI in production. MLOps and infrastructure specialists who size clusters and tune the stack, 24×7, with direct-to-engineer escalation instead of a ticket queue. If a workload needs help moving or scaling, you have people beside you.
  
• Spend you can see. Spend you can see. Autoscaling on VKE, and an idle-killer on managed notebooks that stops billing the moment the work stops, plus minute-level metering and invoices you can split by project…
  
• Ready-to-run models, or bring your own. Serve popular models as token-billed endpoints from the Model Token Factory (in beta), or deploy your own inside your private network.

The savings are real. Take ITQ, one of India’s leading travel-technology companies. It serves fare-rule and refund logic on custom models running on Velocis, around 141 billion tokens a month, and moving to Neysa cut its cost 40% while pulling P99 latency from 14 seconds down to under two. Its CIO, Anil Porter, put it plainly: 

“By partnering with Neysa, we have finally moved past the latency and cost barriers of legacy infrastructure to a system that handles our scale with total reliability.”

ITQ isn’t alone. TIFIN, in wealth management, cut its inference cost 65%; Innoviti, running a multimodal verification engine across a 50,000-merchant network, cut its total cost of ownership 60%.

Cut your inference bill

Inference is the bill that never stops, and it grows with your success. The MI350X makes each token cheaper to serve; Neysa Velocis keeps it fast, reliable and visible as you scale. The only way to know what that does to your numbers is to run it on your own traffic: put one live model on an MI350X, and watch the cost per token.

Talk to a Neysa engineer →

FAQs