Routing, gateways and multi-tenancy

In one paragraph

A model gateway is the single door every model request passes through. It authenticates the caller, applies policy on which models may handle which data, routes each request to a model, falls back when that model is unavailable, enforces budgets, and records what happened. With many teams sharing GPUs, it also isolates them from each other.

How it works

• Routing by task and difficulty. Routine requests go to a small model, harder ones to a larger model. A router can be rules, a classifier, or a learned router such as RouteLLM.
• Residency rules. Requests carrying certain data classes stay on self-hosted models; only approved tasks may reach a hosted model, and those calls are logged.
• Fallbacks and budgets. Each route has a configured fallback and a spending limit.
• Prefix-cache-aware routing. Sending requests with a shared prefix to the replica that already holds it raises cache hits (the Kubernetes Gateway API Inference Extension and llm-d both do this).
• Multi-LoRA serving. One base model serves many adapters, swapped per request.
• Tenant isolation. Separate keys, quotas and logs per team; prefix caches scoped per tenant where timing could leak information.
• Structured outputs enforced at the engine, whatever model answers.

Design choices and trade-offs

When not to use it

A single application calling a single model can start without one, but adding a second model or team is the point to introduce it.

How we evaluate it

Routing accuracy on a labelled sample (did the request go to a model that could handle it), cost and latency per route, and audits that no restricted data class reached an outside route.

Sources

• Ong et al. (2024), RouteLLM (opens in a new tab); Sheng et al. (2023), S-LoRA (opens in a new tab)
• Gateway API Inference Extension (opens in a new tab), llm-d (opens in a new tab)
• Explainer: Model gateways; reference deployment: Hybrid with a gateway