The Self-hosting vs API Decision Is Really an AI Governance Call 

Background

Most enterprises evaluating LLMs are running a cost calculation: is it cheaper to self-host open-weight models or call a hosted API? That is a reasonable question, but it is not the right one. The right one is this: who processes your inference traffic, under what legal framework, and does your AI governance policy actually account for it? 

Those are different questions with very different answers, and which one your team is asking says a lot about where your AI governance actually stands. 

The Prompt Is the Data 

Every time you call a hosted LLM API, your prompt travels to a third-party server, gets processed there, and a response comes back. For most workloads, that works fine. But when those prompts contain patient notes, legal analysis, proprietary trade logic, or client financials, ‘third-party infrastructure’ stops being just a technical description and starts being a compliance category. 

The problem is that most enterprise teams haven’t drawn that line. IBM’s 2026 Cost of a Data Breach Report flagged shadow AI as the leading entry point for enterprise data leakage. 60% of organizations currently have no formal AI security policy governing what happens when a team connects sensitive business data to a public LLM API without IT oversight. That is not a technology gap – it is a governance gap.

The Case That Made This Concrete

In July 2026, Moonshot AI released Kimi K3, the largest open-weight model ever published, at 2.8 trillion parameters. It ranked #3 on the Artificial Analysis Intelligence Index, ahead of Claude Opus 4.8, and first on Arena.ai’s Frontend Code Arena in blind developer voting. A genuinely frontier-class model, available to download and run on your own infrastructure. 

That release forced a decision that had previously been easy to defer. If you access K3 through Moonshot’s hosted API, your prompts land on servers operated by a Beijing-based entity subject to China’s National Intelligence Law, which requires all organizations to cooperate with national intelligence efforts upon request. If you download the weights and run inference on your own infrastructure, your data never reaches Moonshot’s servers at all. That is not a marginal distinction. It is a structural one, and no terms of service agreement can bridge it. 

Three months before K3 launched, the structural risk had already become operational. In April 2026, Kimi’s platform experienced a confirmed cross-user data isolation failure: one user’s full resume, including their name, phone number, and complete work history, was exposed to an unrelated user on the same platform. Moonshot issued no public statement. The OECD AI Incidents Monitor catalogued the event. This wasn’t a theoretical exposure – it was a documented failure on a production API platform with tens of millions of monthly active users, and it is the kind of failure that is significantly more likely to happen on a shared multi-tenant API than on infrastructure you control yourself. 

Some Industries Don’t Get to Choose

For enterprises in healthcare, financial services, and legal, several regulations effectively make the inference-environment decision for you, even if most teams haven’t mapped them through yet. 

HIPAA says, patient data cannot be processed outside your own compliant infrastructure unless the vendor has signed a Business Associate Agreement. Most standard API agreements don’t come with one. GDPR and India’s DPDP Act both treat any third party processing personal data as a formal data processor. Meaning, the documentation, residency requirements, and the consent obligations that most teams haven’t completed for their LLM providers. The EU AI Act reached full enforcement in August 2026, requiring high-risk AI systems to maintain audit trails and human oversight documentation. And across financial services, SOC 2 audits are increasingly flagging AI infrastructure that processes covered data without proper compliance controls. 

The common thread across all of them is that calling a public API with regulated data isn’t a grey area.
It is a gap, and it is one that most enterprise AI deployments haven’t formally closed.

Self-Hosting Has a Long Receipt 

The natural response to all of this is to consider running models on your own infrastructure, and for many regulated workloads, that is exactly the right answer. But the economics are more complicated than the GPU rental price makes them look. 

Running a frontier open-weight model like K3 at production scale requires approximately 1.4 terabytes of fast GPU memory, 64 or more accelerators in a high-bandwidth interconnect domain, and hardware with native MXFP4 support. That is before you account for the operational costs most self-hosting budgets miss: senior engineering talent for maintenance and monitoring (roughly 10 to 20 hours per month at $75 to $150 per hour), model update cycles every six to eight weeks, GPU utilization waste from the bursty nature of enterprise workloads, and hardware depreciation across a 36-month cycle. When you factor all of that in, self-hosting ends up costing three to five times the raw GPU rental price. 

The break-even against frontier API pricing sits around 100 to 256 million tokens per month, a threshold most production systems don’t consistently hit. So self-hosting everything isn’t the answer. The answer is understanding which workloads are sensitive enough that inference must stay inside your network, separating those from workloads where a commercial API is the right and compliant choice, and routing accordingly. 

Velocis: The Piece That Actually Makes This Work 

Routing architecture is straightforward to describe and genuinely hard to operate without the right infrastructure under it. Private cloud endpoints like Azure OpenAI with VNet integration or AWS Bedrock solve the compliance layer for many regulated workloads, though at a 15 to 40% cost premium over direct API access. For workloads where even the managed cloud endpoint isn’t enough, fully self-hosted open-weight models on dedicated GPU infrastructure close the gap, but only if the infrastructure is set up to run those models at economic utilization rates. 

This is the problem Neysa Velocis is purpose-built around. Running frontier open-weight models at the inference layer is about having the interconnect bandwidth, memory architecture, and operational management that keeps utilization high enough for self-hosting to make financial sense. Most enterprise teams trying to build this themselves run into underutilization problems. Add to it the infrastructure maintenance overhead, and model update complexity that quickly erodes the cost advantage. GPU infrastructure that are thoughtful by design – eliminate this kind of operational gap. Which is what makes the compliant self-hosted architecture viable at scale, rather than just technically possible. 

The inference layer is where prompt data is most exposed. But, it isn’t the only place that matters for regulated enterprises. The full governance picture extends to how models are fine-tuned on proprietary data. How outputs are logged and audited, and how AI-driven decisions are documented for regulatory review. That is a conversation about AI data governance more broadly, and Neysa Aegis addresses that layer.

Before You Build Anything, Start Here

If your AI governance policy doesn’t yet have an explicit answer to which inference environments are approved for which data classifications, that is the conversation to have. Because until you do, every team in your organization is making that routing decision on their own. 

Jam with the Neysa team to explore your options and find out what the right architecture looks like for your workloads.