If vLLM already solved LLM serving, why did SGLang appear?
Gartner’s 2025 AI report states that over 85% of ML models never make it to production, and of
those that do, fewer than 40% sustain any meaningful business value beyond 12 months.
Teams tend to pin this on model quality, on data problems, or on a business case that wasn’t
clear enough. The more accurate explanation in most cases, is that nobody could close the gap
between a model that worked in an experiment and one that could handle production traffic. And
by the time that gap closes, the project is usually already out of budget or outside priority.
Time to first deployment (TTFD) is the metric that tells you how wide that gap is; and sadly
most AI teams aren’t measuring it.
TTFD: The Window That Matters
In DORA (DevOps Research and Assessment) terms, this sits closest to change lead time. This measures how long it takes for a code change to move from commit to production. The principle is the same for AI. Just that the path is longer.
Software deployment has a reasonably predictable shape: code gets reviewed, tests pass, the change ships. AI deployment has more steps, and each one takes time – model validation, data pipeline checks, inference testing, GPU provisioning, serving infrastructure setup, monitoring, safety review, and, compliance sign-off. None of these are unreasonable to run. But, in most teams they sit in sequence – meaning total time is the sum of all of them rather than the time it takes to run the slowest one. That’s a solvable problem, but only if someone is measuring it.
Why AI Deployment Takes Longer Than Teams Expect
55% of companies cite the lack of adequate MLOps practices as their main obstacle to deploying ML models, while 70% are actively investing in MLOps tools. The reasons aren’t mysterious once you look at them.
The environment mismatch problem is more painful in AI than in software. When a model trained on one hardware configuration gets served on a different one, you can get genuinely different outputs, not just different latency numbers or memory behavior. That means pre-deployment validation has to check that the model behaves the same way on production hardware as it did during training, which means production hardware needs to be available and ready before deployment even begins. Teams that don’t have this built into their workflow – end up doing manual checks before every deployment. And these checks move only as fast as the person running them.
Then there’s the compute provisioning problem, which is a different monster to tame. A team can have a perfectly clean model, passing validation, all compliance boxes checked, and still wait several days because GPU capacity isn’t available and needs to be provisioned before anything can start. That wait goes directly into the deployment clock, and the deployment clock is what eventually shows up in the business case.
The Four Usual Suspects
When teams measure this properly for the first time, they’re usually surprised by where the time
goes. The most common culprits:
- Compute that isn’t pre-configured. When teams provision GPU clusters fresh for each
deployment, that provisioning time goes into the deployment clock. It’s a separate
problem from model readiness, and mixing the two makes it hard to know which one
you’re actually solving. - Validation steps that run sequentially when they don’t need to. Most teams run model
checks, then set up infrastructure, then run inference tests, then wait for approvals – one
after another. A lot of those steps don’t actually depend on each other and could run at
the same time. They don’t because nobody has mapped the dependency chain carefully
enough to know which handoffs are genuinely blocking and which ones just feel
sequential. - Approval gates stacked at the end. Safety review, compliance, and business sign-off
all have legitimate reasons to exist. When all of them sit in sequence at the final
deployment stage rather than being built into earlier pipeline steps, they add days to a
window that should be short. - Releasing in batches. Teams that hold multiple model versions and deploy them
together get slower cycle times, higher per-deployment risk, and rollbacks that are much
harder to manage. Smaller, separate releases fix all three without adding complexity.
Benchmarks Worth Knowing
DORA’s benchmarks define four bands: elite teams achieve lead times under an hour and
deploy multiple times a day, high performers take between a day and a week, medium
performers one to four weeks, and low performers longer than a month. Most AI teams land in
the medium band, though many don’t actually know which band they’re in because they haven’t
measured it.
For AI specifically, reasonable 2026 targets look something like: first production deployment of a
new model in under a week from model readiness, retraining and fine-tuning updates in hours
rather than days, and rollback in minutes when something breaks in production.
One counter-metric worth tracking alongside deployment speed is change failure rate, because
DORA’s 2025 research shows AI adoption tends to improve throughput while increasing stability
problems when quality checks are reduced to hit speed targets. If deployment time drops but
change failure rate climbs, you’re not getting faster – you’re just moving problems to where
they’re harder to fix.
What Actually Moves the Number
The improvements that consistently work:
- Infrastructure ready before the model is. Pre-configured serving stacks and
on-demand GPU compute mean deployment starts when the model is ready, not after a
provisioning queue clears. - Parallel validation. Map the actual dependencies in your pipeline and run everything
that doesn’t strictly depend on something else at the same time. Model checks, data
pipeline validation, and compliance review can almost always run concurrently. - Smaller releases, not batches. Each model version ships on its own. Less scope per
release means faster feedback and simpler rollback. - Compliance moved upstream. Most approval requirements can be automated earlier in
development so they’re resolved during the build, not blocking the deploy.
The infrastructure piece matters more than most teams expect, and it tends to be the last thing
they look at. Time to first deployment often has less to do with team process, than with what the
underlying compute layer can do. How fast clusters come online, how the serving layer is
configured, whether the deployment pipeline was built for AI workloads or adapted from a
general software stack.
Neysa Velocis is designed around this: GPU infrastructure with pre-configured serving layers sothe gap between model ready and model live is measured in hours, not held back by provisioning queues or environment mismatches.
If your team is looking to cut deployment time without cutting corners on quality, our product
team is happy to walk you through how Velocis is set up to help – start a conversation here.
