# Speculative decoding and disaggregation

> How speculative decoding and disaggregated prefill and decode work, the expected gain, and the load conditions under which they help.

- Web page: https://www.alesanalytics.com/docs/speculative-decoding-and-disaggregation
- Markdown: https://www.alesanalytics.com/docs/speculative-decoding-and-disaggregation.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. Two latency techniques and when they pay. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Speculative decoding lets a cheap proposer guess several tokens that the large model then checks in one pass, so more than one token can be produced per step with no change to the output distribution. Disaggregation puts prefill and decode on separate GPU pools. Both target latency, and both help only under particular conditions.

## How it works

- **Proposers.** A small draft model, extra prediction heads (Medusa), feature-level drafting (EAGLE-3), the model's own multi-token prediction layers, or n-gram lookup from the prompt.
- **Expected gain.** With γ draft tokens and acceptance rate α, expected tokens per step is (1 − α^(γ+1)) ÷ (1 − α). At α = 0.7 and γ = 4, about 2.8.
- **Disaggregated prefill and decode.** Prefill and decode run on different GPUs, with the cache transferred between them, so each phase's latency can be controlled separately. vLLM marks its implementation experimental and states it does not raise throughput.

![Figure 1. One speculative decoding step.](https://www.alesanalytics.com/diagrams/inf-speculative.svg)

## Design choices and trade-offs

- **Load.** Speculation helps most at low to medium load, when decode leaves compute idle; at high batch sizes the gain shrinks or disappears.
- **Acceptance rate.** Depends on the task; n-gram lookup works well when outputs copy from the prompt, as in extraction.
- **Complexity.** Disaggregation adds a cache-transfer path and more components to operate.

## When not to use it

When the deployment is throughput-bound at high concurrency, or when latency already meets its targets.

## How we evaluate it

Measure TTFT, inter-token latency and goodput with and without the technique at the expected concurrency, on the client's prompts.

## Sources

- Leviathan, Kalman and Matias (2023), [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/abs/2211.17192)
- [EAGLE-3](https://arxiv.org/abs/2503.01840), [Medusa](https://arxiv.org/abs/2401.10774)
- [vLLM speculative decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/), [vLLM disaggregated prefill](https://docs.vllm.ai/en/latest/features/disagg_prefill/)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt