# How serving engines schedule work

> Prefill and decode, continuous batching, PagedAttention, chunked prefill, prefix caching and preemption, and the settings that tune them.

- Web page: https://www.alesanalytics.com/docs/how-serving-engines-schedule-work
- Markdown: https://www.alesanalytics.com/docs/how-serving-engines-schedule-work.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. Continuous batching, paged caches and prefix reuse. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Generating text has two phases. Prefill reads the whole prompt in parallel and is limited by compute; decode writes one token at a time and is limited by memory bandwidth. Modern engines are built around that split: they batch requests continuously, store the attention cache in pages, break long prompts into chunks, and reuse cached prefixes.

## How it works

- **Continuous batching.** Requests join and leave the running batch at every decode step, so short answers do not wait for long ones.
- **PagedAttention.** The key-value cache is stored in fixed-size blocks, removing most fragmentation and letting requests share blocks.
- **Chunked prefill.** Long prompts are split and mixed with decode steps, steadying inter-token latency at some cost to the long prompt's time to first token.
- **Prefix caching.** Shared prefixes (system prompts, shared documents) reuse cached blocks; vLLM identifies them by hashing, SGLang organises them in a radix tree.
- **Preemption.** When cache memory runs out, some sequences are paused and their cache swapped out or recomputed.

![Figure 1. Prefill, the paged KV cache and the decode loop.](https://www.alesanalytics.com/diagrams/inf-scheduling.svg)

## Design choices and trade-offs

| Setting                | Trade-off                                                                                      |
| ---------------------- | ---------------------------------------------------------------------------------------------- |
| Chunk size             | Smaller chunks give smoother decoding; larger chunks give faster first tokens for long prompts |
| Maximum batched tokens | Higher improves throughput until latency targets slip                                          |
| Cache memory share     | More cache means more concurrent sequences, less room for activations                          |

## When not to use it

These are engine internals; they matter to whoever sizes and tunes the deployment.

## How we evaluate it

Tune on the client's workload, watching preemption counts, cache hit rate and latency percentiles together.

## Sources

- Kwon et al. (2023), [PagedAttention](https://arxiv.org/abs/2309.06180)
- [vLLM optimisation and tuning](https://docs.vllm.ai/en/latest/configuration/optimization/)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt