# Observability, evaluation and cost

> Tracing with the OpenTelemetry generative AI conventions, quality on a frozen test set and live samples, and cost attributed by team and task.

- Web page: https://www.alesanalytics.com/docs/observability-evaluation-and-cost
- Markdown: https://www.alesanalytics.com/docs/observability-evaluation-and-cost.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. Is it right, is it fast enough, what does it cost. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Once a system is live, three questions need answers every week: is it still right, is it fast enough, and what does it cost? Observability answers them with traces of every request, quality measured on a fixed test set and a sample of live outputs, and cost attributed to the team and task that incurred it.

## How it works

- **Tracing.** Each request is traced from the user through retrieval, model calls and tools. The OpenTelemetry semantic conventions for generative AI define common attributes (model, token counts, latency); they are still marked "Development" and may change.
- **Quality.** The frozen test set runs on every change and on a schedule; a sample of live outputs is reviewed by people each week; review-queue volume and confidence distributions are tracked as early warnings.
- **Performance.** Time to first token, inter-token latency and end-to-end latency at p50, p95 and p99, and queue depth.
- **Cost.** Token counts and GPU time per request, attributed through the gateway to the application, team and task.
- **Alerts.** On regressions against thresholds, not on every fluctuation.

## Design choices and trade-offs

| Choice                                | Trade-off                                                                                                 |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Capturing prompt and response content | Easier debugging and evaluation; privacy and retention obligations. Off by default, sampled with approval |
| Online LLM-as-judge scoring           | Cheap broad coverage; needs calibration against human labels                                              |
| Cost by request vs by GPU hour        | Per-request attribution is fairer for shared GPUs; GPU hours match the bill                               |

## When not to use it

A prototype used by three people for a fortnight needs logs, not a dashboard. Anything in production needs this.

## How we evaluate it

By whether it catches problems: a drill that injects a regression (a degraded model, a changed document layout) and checks that the dashboards and alerts show it before users report it.

## Sources

- [OpenTelemetry semantic conventions for generative AI](https://github.com/open-telemetry/semantic-conventions-genai)
- [Measuring inference performance](https://www.alesanalytics.com/docs/measuring-inference-performance), [Evaluation in CI and production](https://www.alesanalytics.com/docs/evaluation-in-ci-and-production)
- Explainers: [Evaluation sets](https://www.alesanalytics.com/explainers/evaluation-sets), [Total cost of ownership](https://www.alesanalytics.com/explainers/total-cost-of-ownership)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt