# AI Inference and the Model Gateway

> LLM inference explained for architects: prefill and decode, the KV cache, PagedAttention, GPU cost, and what a model gateway must do on every request.

- Web page: https://www.alesanalytics.com/architecture/ai-serving-and-gateways
- Markdown: https://www.alesanalytics.com/architecture/ai-serving-and-gateways.md
- Part of: [Enterprise Architecture Atlas](https://www.alesanalytics.com/architecture.md), section "AI systems"
- Last verified: 13 October 2026
- Publisher: Ales Analytics

## Five subsystems, not one

An enterprise AI system looks like one thing from the chat window. Underneath it is five coupled subsystems, and most failures happen where two of them meet:

- **Serve:** running the model. Prefill and decode, the KV cache, batching, GPUs and how they are scheduled.
- **Access:** the gateway in front of the models. Authentication, authorization, quota, routing and metering.
- **Know:** retrieval. Connectors, chunks with their ACLs and labels, the index, filtering, ranking and verification.
- **Act:** tools and agents. MCP servers, identities, idempotency and approvals.
- **Assure:** evaluation, tracing and red-teaming, for every route and every version.

This page covers the first two. [Retrieval, MCP and agents](https://www.alesanalytics.com/architecture/ai-retrieval-and-agents.md) covers the rest.

## Inference physics

A model reads the whole prompt once, then writes its answer one token at a time. Each new token has to reread the stored keys and values for every token before it. That store is the **KV cache**, and it is why serving a language model is mostly a memory problem.

**Figure: Prefill and decode: the two halves of every model call.** Prefill reads the whole prompt in parallel and builds the KV cache; it is compute-bound and sets time to first token. Decode writes one token at a time, reading the KV cache; it is bound by memory bandwidth and sets inter-token latency. For an illustrative 70B-class model, one 8,000-token sequence needs about 2.6 GB of KV cache, of the roughly 20 GB left on two 80 GB GPUs after FP16 weights, so six or seven fit at once. Prefill reads the whole prompt at once and is limited by compute, so it sets time to first token. Decode writes one token at a time, rereading the KV cache each time, and is limited by memory bandwidth, so it sets the gap between tokens.

| Phase | What it does | What limits it | The metric it sets |
| --- | --- | --- | --- |
| Prefill | Processes the whole prompt in parallel and builds the KV cache | Compute | Time to first token |
| Decode | Produces one token at a time, reading the KV cache | Memory bandwidth | Inter-token latency |

KV-cache memory grows linearly with tokens multiplied by concurrent sequences. The [numbers page](https://www.alesanalytics.com/architecture/numbers.md) works through an illustrative 70B-class model: 320 KiB per token, 2.6 GB for one 8,000-token sequence, and the difference FP8 weights make to how many sequences fit on two GPUs.

### What serving engines do about it

- **PagedAttention.** KV-cache memory is large and changes size constantly, and naive allocation wastes it through fragmentation and duplication, which limits how many requests can be batched. Kwon and colleagues applied operating-system-style paging to the cache, reaching near-zero waste and **2 to 4 times the throughput at the same latency** compared with FasterTransformer and Orca. This is the design behind vLLM.
- **Continuous batching** adds new requests to the batch as others finish, instead of waiting for the whole batch to complete.
- **Prefix caching** reuses the KV cache for a shared opening, such as a long system prompt. It must never be shared across tenants, because the cache is derived from one tenant's prompt.
- **Quantisation** from FP16 to FP8, INT8 or INT4 cuts memory and bandwidth. The quality cost has to be measured per task, not assumed.
- **Parallelism.** Tensor parallelism splits each layer across GPUs and needs NVLink-class interconnects. Pipeline parallelism splits the layers between GPUs. Expert parallelism spreads the experts of a mixture-of-experts model. Disaggregating prefill and decode onto different pools is emerging.

> **A qualification on serving as memory management.**
>
> That holds for decode-heavy, batchable workloads, which is most chat and agent traffic. For long prompts at low concurrency, prefill compute dominates instead. The more accurate version: serving is memory management at batch, and compute management at long prefill.

### Cost: idle GPUs dominate

Self-hosting is only cheaper at high utilisation. AWS now publishes Capacity Block prices per accelerator-hour, and states that they are dynamic. The rates effective 7 October 2026 were:

| Instance family | Capacity Block, $ per accelerator-hour |
| --- | --- |
| P6-B300 | 16.146 |
| P6-B200 | 14.208 |
| P5en | 7.895 |
| P5e | 6.866 |
| P5 | 5.970 |
| P4de | 2.546 |
| P4d | 1.696 |

On-demand and Savings Plans prices were unchanged by that update, and on-demand GPU prices per hour were not verified for this edition. The [cost-per-token calculation](https://www.alesanalytics.com/architecture/numbers.md) shows how a P5 rate turns into about $1.11 per million tokens at full utilisation and $3.68 at 30%.

### How serving fails

KV cache fills, so the engine preempts sequences and latency spikes. GPUs throw Xid errors or run out of memory. Quota errors (HTTP 429) cascade through every caller. Long-context requests starve short ones unless they are pooled separately.

## The model gateway

Every request from an application or an agent should pass through one gateway on its way to a model. It is the single place where you can enforce which data may reach which model, meter what it costs, and fail over safely.

**Figure: What a model gateway does to every request, in order.** 1, authenticate the workload or agent. 2, authorize the model for this class of data. 3, check quota and budget. 4, redact personal data. 5, route. 6, fall back only to an equivalent route. 7, apply the logging policy. 8, meter the tokens. Example routes: Microsoft Foundry Models in an EU deployment, Amazon Bedrock in an EU Region, Gemini Enterprise Agent Platform (formerly Vertex AI). The order is the design. Authorization decides which models this class of data may reach, and the fallback at step 6 may only pick a route that would have passed every check before it.

### The rules that make a gateway safe

**Fallback must be policy-equivalent.** If the primary route fails, the fallback may only be a route that would have passed every earlier check: the same data-class allow-list, the same residency, the same logging. Never fall back to an unapproved model or region. When no equivalent route exists, fail degraded (search results without generation, say), not open.

**A regulatory withdrawal is a routing failure.** A model or region you're no longer permitted to use has to be handled the same way as one that's down.

**Bulkhead per tenant,** so one team's runaway batch job can't exhaust the shared quota.

**Token metering is the gateway's job.** FOCUS 1.4, the FinOps cost-data specification, has no dedicated token columns; FOCUS 1.5 proposes them, with ratification scheduled for 3 December 2026. Until your cost data carries tokens natively, the gateway is where they are counted.

### A fallback is a route, and routes multiply

Treating a fallback model as "the same API, different name" is the most common gateway mistake. Follow the consequences:

- **First order:** the fallback must use the same data-class allow-list, residency and logging.
- **Second order:** the fallback model behaves differently. Its tokenizer changes token counts and costs. Its context window may be smaller, silently truncating the retrieved context and degrading grounding. Its tool-calling schema and structured-output fidelity differ. Its safety policy refuses different things. Its price differs. And the audit trail has to record which model actually answered.
- **Third order:** routes multiplied by prompts multiplied by tools is your evaluation burden. Routing policy is configuration, so it falls under the same validation, canary and rollback discipline as any other configuration.

A gateway is also a shared dependency: Tier-1 at least, and Tier-0 if every AI feature routes through it. It needs to run in more than one region, with a last-known-good configuration, and with bulkheads. And it is one enforcement point among several, not the whole of AI governance: retrieval, tools, data and evaluation live elsewhere and need their own controls.

## Operating it

Trace every request with the prompt-template version, the model version, the retrieved chunk IDs, the tool arguments (redacted), tokens, latency, cost and evaluation scores. Monitor drift. And treat every announced model deprecation as a scheduled outage, with a date, an owner and a tested replacement route.

## Questions to ask in a review

What is KV memory per sequence at our p95 context length? What is utilisation at our latency SLO? What happens when the GPU type we chose isn't available? Which routes are policy-equivalent for each data class, and has each one passed the same evaluation? Who counts the tokens?

Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.

## Sources

1. [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/abs/2309.06180). Kwon et al., 2023. Pages the KV cache as an operating system pages memory. Near-zero waste and 2 to 4 times the throughput at equal latency against FasterTransformer and Orca. The design behind vLLM.
2. [Amazon EC2 Capacity Blocks for ML pricing](https://aws.amazon.com/ec2/capacityblocks/pricing/). Amazon Web Services, 2026. Prices per accelerator-hour, described as dynamic. Used here for the rates effective 7 October 2026.
3. [FOCUS: the FinOps Open Cost and Usage Specification](https://focus.finops.org/). FinOps Foundation, 2026. Version 1.4, ratified 4 June 2026, adds two datasets, 47 columns and six attributes and no dedicated token columns. Token properties are proposed for 1.5.

## Where to go next

- Previous: [Hybrid Cloud, Private Cloud and On-Premises](https://www.alesanalytics.com/architecture/hybrid-and-on-premises.md)
- Next: [Permission-Aware Retrieval, MCP and AI Agents](https://www.alesanalytics.com/architecture/ai-retrieval-and-agents.md)
- Overview and every page: [Enterprise Architecture Atlas](https://www.alesanalytics.com/architecture.md)
- The whole site in one file: https://www.alesanalytics.com/llms-full.txt