# Measuring inference performance

> Which latency and throughput measures matter, why percentiles and goodput beat averages, and how to make a benchmark realistic.

- Web page: https://www.alesanalytics.com/docs/measuring-inference-performance
- Markdown: https://www.alesanalytics.com/docs/measuring-inference-performance.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. TTFT, inter-token latency and goodput. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Inference performance is measured from the user's side: how long until the first token appears, how smoothly the rest streams, and how long the whole answer takes, at the load the system will really carry. Throughput alone hides whether users are waiting, so the headline measure is goodput, the highest request rate at which a stated share of requests meet the latency targets.

## How it works

| Measure                    | Meaning                                                                            |
| -------------------------- | ---------------------------------------------------------------------------------- |
| Time to first token (TTFT) | Queueing plus prefill                                                              |
| Inter-token latency (ITL)  | Time between streamed tokens; decode speed                                         |
| End-to-end latency         | Whole response                                                                     |
| Throughput                 | Tokens per second across all users                                                 |
| Goodput                    | Highest rate at which, for example, 95% of requests meet both TTFT and ITL targets |

Report p50, p95 and p99, not averages.

## Design choices and trade-offs

- **Workload realism.** Prompt and output lengths drawn from real traffic, and the real degree of prefix sharing. Identical prompts overstate a system with prefix caching; random prompts understate it.
- **Open or closed load.** A fixed arrival rate (open loop) shows queueing; a fixed number of users (closed loop) does not.
- **Tools.** GuideLLM and vLLM's benchmark suite are common options.

## When not to use it

Batch jobs that run overnight care about total time and cost, not TTFT.

## How we evaluate it

Every reported number states hardware, engine version, model and quantisation, prompt distribution, concurrency and the targets used.

## Sources

- [GuideLLM](https://github.com/vllm-project/guidellm)
- [Inference performance](https://www.alesanalytics.com/engineering/inference), which defines goodput with a worked example

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt