# Routing, gateways and multi-tenancy

> How a model gateway routes by task and data class, enforces budgets and fallbacks, serves many adapters and isolates tenants.

- Web page: https://www.alesanalytics.com/docs/routing-gateways-and-multi-tenancy
- Markdown: https://www.alesanalytics.com/docs/routing-gateways-and-multi-tenancy.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. One door for every model request. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

A model gateway is the single door every model request passes through. It authenticates the caller, applies policy on which models may handle which data, routes each request to a model, falls back when that model is unavailable, enforces budgets, and records what happened. With many teams sharing GPUs, it also isolates them from each other.

## How it works

- **Routing by task and difficulty.** Routine requests go to a small model, harder ones to a larger model. A router can be rules, a classifier, or a learned router such as RouteLLM.
- **Residency rules.** Requests carrying certain data classes stay on self-hosted models; only approved tasks may reach a hosted model, and those calls are logged.
- **Fallbacks and budgets.** Each route has a configured fallback and a spending limit.
- **Prefix-cache-aware routing.** Sending requests with a shared prefix to the replica that already holds it raises cache hits (the Kubernetes Gateway API Inference Extension and llm-d both do this).
- **Multi-LoRA serving.** One base model serves many adapters, swapped per request.
- **Tenant isolation.** Separate keys, quotas and logs per team; prefix caches scoped per tenant where timing could leak information.
- **Structured outputs** enforced at the engine, whatever model answers.

![Figure 1. Routing by task and data class through one gateway.](https://www.alesanalytics.com/diagrams/inf-gateway.svg)

## Design choices and trade-offs

| Choice                                | Trade-off                                                                                                          |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| Open-source gateway library vs custom | Faster start vs full control; either way, pin and verify dependencies, as the March 2026 LiteLLM compromise showed |
| Learned router vs rules               | Learned routers save more; rules are easier to audit                                                               |

## When not to use it

A single application calling a single model can start without one, but adding a second model or team is the point to introduce it.

## How we evaluate it

Routing accuracy on a labelled sample (did the request go to a model that could handle it), cost and latency per route, and audits that no restricted data class reached an outside route.

## Sources

- Ong et al. (2024), [RouteLLM](https://arxiv.org/abs/2406.18665); Sheng et al. (2023), [S-LoRA](https://arxiv.org/abs/2311.03285)
- [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/), [llm-d](https://llm-d.ai/)
- Explainer: [Model gateways](https://www.alesanalytics.com/explainers/model-gateway); reference deployment: [Hybrid with a gateway](https://www.alesanalytics.com/docs/hybrid-with-a-gateway)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt