Evaluation in CI and production

In one paragraph

Evaluation does not stop at launch. In continuous integration, every change to a prompt, model, adapter or retrieval setting runs the frozen test set, and a regression beyond an agreed threshold blocks the release. In production, new versions run in shadow and then on a small share of traffic, drift signals are watched, a sample of outputs is reviewed by people, and rollback is a configuration change.

How it works

• Regression gates. Thresholds per metric and per segment (document type, field, question type), not only an overall score.
• Shadow releases. The candidate receives real traffic and logs outputs without serving them; outputs are compared with the live version.
• Canary releases. The candidate serves a small share of traffic, widened as metrics hold.
• Drift signals. Changes in input mix, confidence distributions and review-queue volume, which usually move before accuracy visibly drops.
• Sampled review. A fixed weekly sample reviewed by people; disagreements are added to the test set as a new version.
• Rollback. The previous model, adapter, prompt and index stay deployable.

Design choices and trade-offs

When not to use it

A system that never changes would not need it, but models, documents and rules all change.

How we evaluate it

By drills: deliberately ship a degraded candidate to a staging environment and confirm the gate blocks it.

Sources

• Model adaptation and evaluation
• Observability, evaluation and cost