# Inference performance

> How modern serving engines batch and cache, when speculative decoding helps, how quantisation trades memory for accuracy, and how to size a deployment.

- Web page: https://www.alesanalytics.com/engineering/inference
- Markdown: https://www.alesanalytics.com/engineering/inference.md
- Last updated: 2026-11-07
- Publisher: Ales Analytics

Serving engines, scheduling, quantisation and capacity sizing. Published 2026-11-07. Last reviewed 2026-11-07. Reviewed by Ales Analytics engineering.

## Choosing a serving engine

A serving engine turns model weights on a GPU into an API that many people can call at once. The engine is chosen for the hardware, the model architecture and the team that will operate it. There is no single ranking, because published benchmarks differ in hardware, model, prompt lengths and settings, and each engine wins somewhere.

| Engine               | Typical fit                                                                              |
| -------------------- | ---------------------------------------------------------------------------------------- |
| vLLM                 | General-purpose GPU serving with broad model support and an OpenAI-compatible API        |
| SGLang               | Workloads with heavy prefix sharing or structured generation, using its radix-tree cache |
| TensorRT-LLM         | NVIDIA hardware where compiled, hardware-specific kernels justify a build step           |
| LMDeploy             | An alternative GPU engine with its own TurboMind backend and strong quantisation support |
| llama.cpp, Ollama    | Small deployments, CPUs, laptops and edge devices, often with GGUF quantised weights     |
| NVIDIA Dynamo, llm-d | Cluster layers above an engine: routing, disaggregation and scaling across many GPUs     |

A fair comparison runs the client's own prompts, at the client's own concurrency, on the target hardware, and reports the measures described under [Measuring performance](#measuring-performance).

## How a modern engine schedules work

Generating a response has two phases with very different costs.

- **Prefill** processes the whole prompt in one pass. Many tokens are computed in parallel, so it is compute-bound and sets the time to first token.
- **Decode** produces one token at a time per sequence. Each step reads all the model weights to produce very little arithmetic, so it is bound by memory bandwidth (see the worked example below).

The techniques that make an engine efficient follow from that split.

- **Continuous batching.** Instead of waiting for a whole batch to finish, the scheduler adds new requests and removes finished ones at every decode step, so the GPU never idles while a long answer finishes.
- **PagedAttention.** The attention key-value (KV) cache is stored in fixed-size blocks, like pages in virtual memory, rather than in one contiguous allocation per request. This removes most of the fragmentation that wastes cache memory and lets requests share blocks.
- **Chunked prefill.** A long prompt is split into chunks and mixed into decode batches. Decoding users see steadier inter-token latency, while the long prompt waits a little longer for its first token. The chunk size sets the trade-off.
- **Prefix caching.** Requests that share a prefix (the same system prompt, the same document) reuse its cached KV blocks. vLLM identifies blocks by hashing their contents; SGLang organises them in a radix tree.
- **Preemption.** When cache memory runs out, the scheduler pauses some sequences, either swapping their cache out or recomputing it later. Frequent preemption is a sign the deployment is undersized for its load.

![Figure 1. How a serving engine schedules work: prefill fills a paged KV cache, and each decode step reads it and adds to it.](https://www.alesanalytics.com/diagrams/inf-scheduling.svg)

## Latency techniques

- **Speculative decoding.** A cheap proposer guesses several next tokens, and the large model verifies them in a single forward pass, keeping the ones it agrees with. The output distribution is unchanged. Proposers include a small draft model, extra prediction heads (Medusa), feature-level drafting (EAGLE-3), the model's own multi-token prediction layers, and n-gram lookup from the prompt. It helps most at low to medium load, when the GPU has spare compute during decode.
- **Disaggregated prefill and decode.** Prefill and decode run on separate GPU pools so each can be tuned for its phase, with the KV cache transferred between them. vLLM marks its implementation as experimental and states that it does not raise throughput; its use is in controlling tail latency for each phase separately.

## Quantisation

Quantisation stores numbers in fewer bits. It reduces memory and bandwidth, and on hardware with matching tensor cores it can also speed up the arithmetic.

| Approach              | What is quantised                                   | Notes                                                                                            |
| --------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| AWQ, GPTQ             | Weights only, typically to 4 bits                   | Activations stay in 16-bit; the main gain is memory and decode bandwidth                         |
| INT8, FP8             | Weights and activations                             | FP8 is supported natively on Hopper-class and newer GPUs                                         |
| NVFP4, MXFP4          | 4-bit floating point with small-block scale factors | NVFP4 uses blocks of 16 values with FP8 scales; MXFP4 uses blocks of 32 with power-of-two scales |
| KV-cache quantisation | The attention cache, often to FP8                   | Raises the number of concurrent sequences a GPU can hold                                         |

Every quantised model is evaluated against its full-precision baseline on the client's own task set before it serves traffic. Average benchmark scores often hide regressions on specific fields, long documents or numerical content.

## Sizing a deployment

Memory is usually the first constraint. Three quantities decide it.

```
weight memory          = parameters × bytes per parameter
KV cache per token     = 2 × layers × KV heads × head dimension × bytes per value
total KV cache         = KV cache per token × context length × concurrent sequences
```

Add headroom for activations, the CUDA context and fragmentation, typically 10 to 20% of GPU memory. If the result does not fit one GPU, the model is split:

- **Tensor parallelism** splits each layer's matrices across GPUs in one server, with fast interconnect between them.
- **Pipeline parallelism** places different layers on different GPUs or servers.
- **Expert parallelism** places different experts of a mixture-of-experts model on different GPUs.
- **Data parallelism** runs independent replicas to serve more users.

See the [sizing worked example](#sizing-the-kv-cache) below.

## Measuring performance

| Measure                    | What it tells you                                                                |
| -------------------------- | -------------------------------------------------------------------------------- |
| Time to first token (TTFT) | How long a user waits before anything appears; dominated by queueing and prefill |
| Inter-token latency (ITL)  | How smoothly text streams; dominated by decode                                   |
| End-to-end latency         | Total time for a complete response                                               |
| Throughput                 | Tokens per second across all users                                               |
| Goodput                    | The request rate at which a stated share of requests meet both latency targets   |

Report percentiles (p50, p95, p99), not averages. Use realistic prompt lengths and output lengths drawn from the actual workload, and the real degree of prefix sharing: a benchmark with no shared prefix understates a system with a long common system prompt, and one with identical prompts overstates it. GuideLLM and vLLM's own benchmark suite are common tools.

## Routing and multi-tenancy

- **Routing by difficulty.** A router sends routine requests to a small model and harder ones to a larger model, trained or tuned on the client's own examples. RouteLLM is one published approach.
- **Fallbacks.** Every route has a configured fallback for when its first choice is unavailable or over budget.
- **Prefix-cache-aware routing.** Sending requests that share a prefix to the replica already holding it raises cache hit rates. The Kubernetes Gateway API Inference Extension and llm-d both implement this kind of routing.
- **Residency rules.** The gateway decides which requests may reach a hosted model at all, based on the data they carry, and logs every request that does.
- **Multi-LoRA serving.** One base model serves many adapters, loaded and swapped per request, as in S-LoRA and vLLM's LoRA support.
- **Per-tenant cache isolation.** Prefix caches are scoped per tenant where timing differences between cached and uncached prompts could reveal what another tenant asked.
- **Structured outputs.** Grammar-constrained decoding is enforced at the engine, so every response matches its schema whichever model produced it.

![Figure 4. Routing through a gateway by task and data class, with fallbacks and every request logged.](https://www.alesanalytics.com/diagrams/inf-gateway.svg)

## Worked examples

### Sizing the KV cache

Illustrative example. This explains a technique; it does not describe any client system. The model shape is hypothetical.

Take a hypothetical 8-billion-parameter model with grouped-query attention: 32 layers, 8 KV heads, head dimension 128, cache stored in 16-bit (2 bytes).

```
KV cache per token = 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB
16 sequences × 8,192 tokens          = 131,072 tokens in cache
131,072 tokens × 128 KiB             = 16 GiB of KV cache
weights: 8 × 10⁹ parameters × 2 bytes ≈ 16 GB
```

So this deployment needs roughly 32 GB before headroom. Storing the cache in FP8 halves it to 8 GiB, which is why cache precision is a capacity decision as much as an accuracy one. Doubling the context length doubles the cache; so does doubling the number of concurrent users.

![Figure 2. The memory budget from the worked example, in GB: weights and KV cache are about equal, before headroom.](https://www.alesanalytics.com/diagrams/inf-memory.svg)

### Why single-stream decode is bandwidth-bound

Illustrative example. This explains a technique; it does not describe any client system.

Each new token requires reading every weight once. At batch size 1, a model with 16 GB of weights on a GPU with 3.35 TB/s of memory bandwidth (NVIDIA's published figure for the H100 SXM) cannot do better than

```
16 GB ÷ 3,350 GB/s ≈ 4.8 ms per token  ≈ 210 tokens per second
```

and that is before reading the KV cache. Batching shares each weight read across many sequences, which is why total throughput rises with concurrency until compute or cache memory runs out.

### Little's law for capacity planning

Illustrative example. This explains a technique; it does not describe any client system.

```
requests in flight ≈ arrival rate × time in system
4 requests per second × 6 seconds per request ≈ 24 requests in flight
```

The KV cache budget must hold all 24 at the target context length. If each is allowed 8,192 tokens, that is 196,608 tokens of cache, which with the model above is 24 GiB in 16-bit.

### When speculative decoding pays

Illustrative example. This explains a technique; it does not describe any client system.

With γ draft tokens per step and an average acceptance rate α, the expected number of tokens produced per verification step is (Leviathan et al., 2023):

```
E[tokens per step] = (1 − α^(γ+1)) ÷ (1 − α)
α = 0.7, γ = 4  →  (1 − 0.7⁵) ÷ 0.3 ≈ 2.8
```

That is the gain in the best case. Each step also pays for the draft and for verifying γ tokens, and as batch size grows the GPU stops being idle between decode steps, so the speed-up shrinks and can disappear. It is measured at the expected concurrency, not at batch size 1.

![Figure 3. One speculative step: four proposed tokens verified in a single target pass, three accepted, one corrected.](https://www.alesanalytics.com/diagrams/inf-speculative.svg)

### Goodput, not throughput

Illustrative example. This explains a technique; it does not describe any client system.

Define the targets first, for example TTFT under 1 second and inter-token latency under 50 ms. Then increase the request rate step by step and record, at each rate, the share of requests meeting both. Goodput is the highest rate at which, say, 95% of requests still meet both targets. Raw tokens per second can keep rising well past that point while most users wait too long.

## Sources

- [vLLM features](https://docs.vllm.ai/en/latest/features/), [speculative decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/), [disaggregated prefill](https://docs.vllm.ai/en/latest/features/disagg_prefill/), [optimisation and tuning](https://docs.vllm.ai/en/latest/configuration/optimization/)
- [SGLang](https://docs.sglang.io/), [TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/latest/overview.html), [LMDeploy](https://github.com/InternLM/lmdeploy), [llama.cpp](https://github.com/ggml-org/llama.cpp)
- [NVIDIA Dynamo](https://docs.nvidia.com/dynamo/latest/), [llm-d](https://llm-d.ai/), [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/)
- [GuideLLM](https://github.com/vllm-project/guidellm)
- Kwon et al. (2023), [Efficient Memory Management for LLM Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
- Leviathan, Kalman and Matias (2023), [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192)
- Li et al. (2025), [EAGLE-3](https://arxiv.org/abs/2503.01840); Cai et al. (2024), [Medusa](https://arxiv.org/abs/2401.10774)
- Lin et al. (2023), [AWQ](https://arxiv.org/abs/2306.00978); Frantar et al. (2022), [GPTQ](https://arxiv.org/abs/2210.17323)
- [NVIDIA: introducing NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)
- Sheng et al. (2023), [S-LoRA](https://arxiv.org/abs/2311.03285); Ong et al. (2024), [RouteLLM](https://arxiv.org/abs/2406.18665)
- [NVIDIA H100 specifications](https://www.nvidia.com/en-us/data-center/h100/)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt