AI Inference and the Model Gateway

Five subsystems, not one

An enterprise AI system looks like one thing from the chat window. Underneath it is five coupled subsystems, and most failures happen where two of them meet:

Serve: running the model. Prefill and decode, the KV cache, batching, GPUs and how they are scheduled.
Access: the gateway in front of the models. Authentication, authorization, quota, routing and metering.
Know: retrieval. Connectors, chunks with their ACLs and labels, the index, filtering, ranking and verification.
Act: tools and agents. MCP servers, identities, idempotency and approvals.
Assure: evaluation, tracing and red-teaming, for every route and every version.

This page covers the first two. Retrieval, MCP and agents covers the rest.

Inference physics

A model reads the whole prompt once, then writes its answer one token at a time. Each new token has to reread the stored keys and values for every token before it. That store is the , and it is why serving a language model is mostly a memory problem.

Prefill and decode: the two halves of every model call

Prefill

Reads the whole prompt in parallel and builds the KV cache. Compute-bound. Sets time to first token

Decode

Writes one token at a time, reading the KV cache. Bound by memory bandwidth. Sets inter-token latency

KV cache for one 8,000-token sequence, illustrative 70B-class model

2.6 GB

of about 20 GB left on two 80 GB GPUs after FP16 weights: six or seven at once

Prefill reads the whole prompt at once and is limited by compute, so it sets time to first token. Decode writes one token at a time, rereading the KV cache each time, and is limited by memory bandwidth, so it sets the gap between tokens.

Phase	What it does	What limits it	The metric it sets
Prefill	Processes the whole prompt in parallel and builds the KV cache	Compute
Decode	Produces one token at a time, reading the KV cache	Memory bandwidth	Inter-token latency

KV-cache memory grows linearly with tokens multiplied by concurrent sequences. The numbers page works through an illustrative 70B-class model: 320 KiB per token, 2.6 GB for one 8,000-token sequence, and the difference FP8 weights make to how many sequences fit on two GPUs.

What serving engines do about it

PagedAttention. KV-cache memory is large and changes size constantly, and naive allocation wastes it through fragmentation and duplication, which limits how many requests can be batched. Kwon and colleagues applied operating-system-style paging to the cache, reaching near-zero waste and 2 to 4 times the throughput at the same latency compared with FasterTransformer and Orca. This is the design behind vLLM.
Continuous batching adds new requests to the batch as others finish, instead of waiting for the whole batch to complete.
Prefix caching reuses the KV cache for a shared opening, such as a long system prompt. It must never be shared across tenants, because the cache is derived from one tenant's prompt.
Quantisation from FP16 to FP8, INT8 or INT4 cuts memory and bandwidth. The quality cost has to be measured per task, not assumed.
Parallelism. Tensor parallelism splits each layer across GPUs and needs NVLink-class interconnects. Pipeline parallelism splits the layers between GPUs. Expert parallelism spreads the experts of a mixture-of-experts model. Disaggregating prefill and decode onto different pools is emerging.

Cost: idle GPUs dominate

Self-hosting is only cheaper at high utilisation. AWS now publishes Capacity Block prices per accelerator-hour, and states that they are dynamic. The rates effective 7 October 2026 were:

Instance family	Capacity Block, $ per accelerator-hour
P6-B300	16.146
P6-B200	14.208
P5en	7.895
P5e	6.866
P5	5.970
P4de	2.546
P4d	1.696

On-demand and Savings Plans prices were unchanged by that update, and on-demand GPU prices per hour were not verified for this edition. The cost-per-token calculation shows how a P5 rate turns into about $1.11 per million tokens at full utilisation and $3.68 at 30%.

How serving fails

KV cache fills, so the engine preempts sequences and latency spikes. GPUs throw Xid errors or run out of memory. Quota errors (HTTP 429) cascade through every caller. Long-context requests starve short ones unless they are pooled separately.

The model gateway

Every request from an application or an agent should pass through one gateway on its way to a model. It is the single place where you can enforce which data may reach which model, meter what it costs, and fail over safely.

What a model gateway does to every request, in order

Model gateway

1.Authenticate the workload or agent
2.Authorize the model for this data class
3.Check quota and budget
4.Redact personal data
5.Route
6.Fall back only to an equivalent route
7.Apply the logging policy
8.Meter the tokens

Example routes

Microsoft Foundry Models

EU deployment

Amazon Bedrock

EU Region

Gemini Enterprise Agent Platform

Formerly Vertex AI

The order is the design. Authorization decides which models this class of data may reach, and the fallback at step 6 may only pick a route that would have passed every check before it.

The rules that make a gateway safe

Fallback must be policy-equivalent. If the primary route fails, the fallback may only be a route that would have passed every earlier check: the same data-class allow-list, the same residency, the same logging. Never fall back to an unapproved model or region. When no equivalent route exists, fail degraded (search results without generation, say), not open.

A regulatory withdrawal is a routing failure. A model or region you're no longer permitted to use has to be handled the same way as one that's down.

Bulkhead per tenant, so one team's runaway batch job can't exhaust the shared quota.

Token metering is the gateway's job. FOCUS 1.4, the FinOps cost-data specification, has no dedicated token columns; FOCUS 1.5 proposes them, with ratification scheduled for 3 December 2026. Until your cost data carries tokens natively, the gateway is where they are counted.

A fallback is a route, and routes multiply

Treating a fallback model as "the same API, different name" is the most common gateway mistake. Follow the consequences:

First order: the fallback must use the same data-class allow-list, residency and logging.
Second order: the fallback model behaves differently. Its tokenizer changes token counts and costs. Its context window may be smaller, silently truncating the retrieved context and degrading grounding. Its tool-calling schema and structured-output fidelity differ. Its safety policy refuses different things. Its price differs. And the audit trail has to record which model actually answered.
Third order: routes multiplied by prompts multiplied by tools is your evaluation burden. Routing policy is configuration, so it falls under the same validation, canary and rollback discipline as any other configuration.

A gateway is also a shared dependency: Tier-1 at least, and Tier-0 if every AI feature routes through it. It needs to run in more than one region, with a last-known-good configuration, and with bulkheads. And it is one enforcement point among several, not the whole of AI governance: retrieval, tools, data and evaluation live elsewhere and need their own controls.

Operating it

Trace every request with the prompt-template version, the model version, the retrieved chunk IDs, the tool arguments (redacted), tokens, latency, cost and evaluation scores. Monitor drift. And treat every announced model deprecation as a scheduled outage, with a date, an owner and a tested replacement route.

Questions to ask in a review

What is KV memory per sequence at our p95 context length? What is utilisation at our latency SLO? What happens when the GPU type we chose isn't available? Which routes are policy-equivalent for each data class, and has each one passed the same evaluation? Who counts the tokens?