# Evaluation in CI and production

> How every change is gated on the frozen test set, released through shadow and canary stages, watched for drift and rolled back.

- Web page: https://www.alesanalytics.com/docs/evaluation-in-ci-and-production
- Markdown: https://www.alesanalytics.com/docs/evaluation-in-ci-and-production.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. Regression gates, canaries, drift and rollback. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Evaluation does not stop at launch. In continuous integration, every change to a prompt, model, adapter or retrieval setting runs the frozen test set, and a regression beyond an agreed threshold blocks the release. In production, new versions run in shadow and then on a small share of traffic, drift signals are watched, a sample of outputs is reviewed by people, and rollback is a configuration change.

## How it works

- **Regression gates.** Thresholds per metric and per segment (document type, field, question type), not only an overall score.
- **Shadow releases.** The candidate receives real traffic and logs outputs without serving them; outputs are compared with the live version.
- **Canary releases.** The candidate serves a small share of traffic, widened as metrics hold.
- **Drift signals.** Changes in input mix, confidence distributions and review-queue volume, which usually move before accuracy visibly drops.
- **Sampled review.** A fixed weekly sample reviewed by people; disagreements are added to the test set as a new version.
- **Rollback.** The previous model, adapter, prompt and index stay deployable.

![Figure 1. From change to release, with rollback.](https://www.alesanalytics.com/diagrams/ma-release.svg)

## Design choices and trade-offs

| Choice                      | Trade-off                                                                                                              |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Strict vs loose thresholds  | Strict thresholds block noise as well as regressions; thresholds are set from the confidence intervals of the test set |
| Shadow traffic with content | Best comparison; must follow the same retention rules as production logs                                               |

## When not to use it

A system that never changes would not need it, but models, documents and rules all change.

## How we evaluate it

By drills: deliberately ship a degraded candidate to a staging environment and confirm the gate blocks it.

## Sources

- [Model adaptation and evaluation](https://www.alesanalytics.com/engineering/model-adaptation)
- [Observability, evaluation and cost](https://www.alesanalytics.com/docs/observability-evaluation-and-cost)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt