Measuring inference performance

In one paragraph

Inference performance is measured from the user's side: how long until the first token appears, how smoothly the rest streams, and how long the whole answer takes, at the load the system will really carry. Throughput alone hides whether users are waiting, so the headline measure is goodput, the highest request rate at which a stated share of requests meet the latency targets.

How it works

Report p50, p95 and p99, not averages.

Design choices and trade-offs

• Workload realism. Prompt and output lengths drawn from real traffic, and the real degree of prefix sharing. Identical prompts overstate a system with prefix caching; random prompts understate it.
• Open or closed load. A fixed arrival rate (open loop) shows queueing; a fixed number of users (closed loop) does not.
• Tools. GuideLLM and vLLM's benchmark suite are common options.

When not to use it

Batch jobs that run overnight care about total time and cost, not TTFT.

How we evaluate it

Every reported number states hardware, engine version, model and quantisation, prompt distribution, concurrency and the targets used.

Sources

• GuideLLM (opens in a new tab)
• Inference performance, which defines goodput with a worked example