Table of contents
In this article
Technical page · Serving
How serving engines schedule work
Continuous batching, paged caches and prefix reuse
- Published
- 8 November 2026
- Last reviewed
- 8 November 2026
- Re-checked every
- 90 days
- Reviewed by
- Ales Analytics engineering
In one paragraph
Generating text has two phases. Prefill reads the whole prompt in parallel and is limited by compute; decode writes one token at a time and is limited by memory bandwidth. Modern engines are built around that split: they batch requests continuously, store the attention cache in pages, break long prompts into chunks, and reuse cached prefixes.
How it works
- Continuous batching. Requests join and leave the running batch at every decode step, so short answers do not wait for long ones.
- PagedAttention. The key-value cache is stored in fixed-size blocks, removing most fragmentation and letting requests share blocks.
- Chunked prefill. Long prompts are split and mixed with decode steps, steadying inter-token latency at some cost to the long prompt's time to first token.
- Prefix caching. Shared prefixes (system prompts, shared documents) reuse cached blocks; vLLM identifies them by hashing, SGLang organises them in a radix tree.
- Preemption. When cache memory runs out, some sequences are paused and their cache swapped out or recomputed.
Design choices and trade-offs
| Setting | Trade-off |
|---|---|
| Chunk size | Smaller chunks give smoother decoding; larger chunks give faster first tokens for long prompts |
| Maximum batched tokens | Higher improves throughput until latency targets slip |
| Cache memory share | More cache means more concurrent sequences, less room for activations |
When not to use it
These are engine internals; they matter to whoever sizes and tunes the deployment.
How we evaluate it
Tune on the client's workload, watching preemption counts, cache hit rate and latency percentiles together.
Sources
Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.