Observability, evaluation and cost

In one paragraph

Once a system is live, three questions need answers every week: is it still right, is it fast enough, and what does it cost? Observability answers them with traces of every request, quality measured on a fixed test set and a sample of live outputs, and cost attributed to the team and task that incurred it.

How it works

• Tracing. Each request is traced from the user through retrieval, model calls and tools. The OpenTelemetry semantic conventions for generative AI define common attributes (model, token counts, latency); they are still marked "Development" and may change.
• Quality. The frozen test set runs on every change and on a schedule; a sample of live outputs is reviewed by people each week; review-queue volume and confidence distributions are tracked as early warnings.
• Performance. Time to first token, inter-token latency and end-to-end latency at p50, p95 and p99, and queue depth.
• Cost. Token counts and GPU time per request, attributed through the gateway to the application, team and task.
• Alerts. On regressions against thresholds, not on every fluctuation.

Design choices and trade-offs

When not to use it

A prototype used by three people for a fortnight needs logs, not a dashboard. Anything in production needs this.

How we evaluate it

By whether it catches problems: a drill that injects a regression (a degraded model, a changed document layout) and checks that the dashboards and alerts show it before users report it.

Sources

• OpenTelemetry semantic conventions for generative AI (opens in a new tab)
• Measuring inference performance, Evaluation in CI and production
• Explainers: Evaluation sets, Total cost of ownership