If vLLM already solved LLM serving, why did SGLang appear?
Summary
Simple infrastructure doesn’t mean compromised capability. An educational AI platform needed to serve 500 learners across two open-weight models – text and vision-language, on a single 48 GB GPU. Rather than layering on gateways, load balancers and databases, the team built something leaner: one FastAPI service, file-based API key management, and model co-residency in vLLM. The result: production stability, auditable access control, and low operational overhead for a small team to run and recover.
The challenge
A Generative AI education provider faced competing constraints. 500 learners, teaching assistants and faculty needed shared, internet-accessible access to two open-weight models across an active curriculum. Roughly 20 concurrent users at any time would spike around deadlines and live sessions. Every user needed traceable identity and on-demand key revocation. The entire stack had to fit on one 48 GB GPU, maintain sub-10-second latency for live use, and stay simple enough for a small ops team to troubleshoot and recover without vendor tools or database migrations.
The solution
Instead of stacking components, the team reduced surface area.
A single FastAPI gateway handles authentication and routing – validating bearer tokens in-memory against a plain-text key file, no database call on the request path. Both quantized models run co-resident in vLLM, exposed as two OpenAI-compatible endpoints. A small admin script manages the full key lifecycle: bulk provisioning for all 500 users before programme start, instant rotation or revocation, and emergency kill-switch recovery via file restoration.
Context length is capped at 8,192 tokens to keep the KV cache in memory.
Existing client libraries work with only a base URL and key change. There is no proxy to terminate connections, no migrations to manage, no external dependencies beyond vLLM.
Outcome
Speed and Simplicity
Spin up GPU-powered workloads in minutes. Reduce time-to-result with low-latency access and optimized pipelines that simplify everything from provisioning to inference. No multi-dashboard sprawl, no vendor lock-in.
Security and Control
Enterprise-grade, multi-layered security designed for regulated environments. Role-based access, policy enforcement, real-time audit trails, and zero-trust architecture — built in.
Scalability
From 1 to 1,000+ GPUs, scale as your models grow. Neysa Velocis is architected to support elastic scaling, hybrid burst, and workload-aware optimization.
Cost-Effectiveness
Stop burning cycles on idle infra. Transparent billing, spot-aware scheduling, and multi-tiered GPU pricing deliver superior tokens-per-dollar efficiency.
Solving for LLM at Scale for Fast API Gateways
To learn more about the Neysa solutions featured in this story, please contact your Neysa representative
