Speculative decoding and disaggregation

In one paragraph

Speculative decoding lets a cheap proposer guess several tokens that the large model then checks in one pass, so more than one token can be produced per step with no change to the output distribution. Disaggregation puts prefill and decode on separate GPU pools. Both target latency, and both help only under particular conditions.

How it works

• Proposers. A small draft model, extra prediction heads (Medusa), feature-level drafting (EAGLE-3), the model's own multi-token prediction layers, or n-gram lookup from the prompt.
• Expected gain. With γ draft tokens and acceptance rate α, expected tokens per step is (1 − α^(γ+1)) ÷ (1 − α). At α = 0.7 and γ = 4, about 2.8.
• Disaggregated prefill and decode. Prefill and decode run on different GPUs, with the cache transferred between them, so each phase's latency can be controlled separately. vLLM marks its implementation experimental and states it does not raise throughput.

Design choices and trade-offs

• Load. Speculation helps most at low to medium load, when decode leaves compute idle; at high batch sizes the gain shrinks or disappears.
• Acceptance rate. Depends on the task; n-gram lookup works well when outputs copy from the prompt, as in extraction.
• Complexity. Disaggregation adds a cache-transfer path and more components to operate.

When not to use it

When the deployment is throughput-bound at high concurrency, or when latency already meets its targets.

How we evaluate it

Measure TTFT, inter-token latency and goodput with and without the technique at the expected concurrency, on the client's prompts.

Sources

• Leviathan, Kalman and Matias (2023), Fast Inference from Transformers via Speculative Decoding (opens in a new tab)
• EAGLE-3 (opens in a new tab), Medusa (opens in a new tab)
• vLLM speculative decoding (opens in a new tab), vLLM disaggregated prefill (opens in a new tab)