Table of contents
In this article
Technical page · Models
Evaluation built from your own data
A frozen test set labelled by your own experts
- Published
- 8 November 2026
- Last reviewed
- 8 November 2026
- Re-checked every
- 180 days
- Reviewed by
- Ales Analytics engineering
In one paragraph
An evaluation set built from the client's own data is a frozen collection of real inputs (documents, questions, cases) with correct outputs labelled by the client's own experts. It is the only reliable way to know whether a system works on the work it was built for, and every change to the system is scored against it.
How it works
- Sample. Draw real inputs covering the mix of types, layouts, sources and difficulty the system will see, including awkward cases.
- Label. The client's experts label the correct output. Two labellers on a subset show how much people themselves disagree.
- Freeze and version. The set is never used for training or prompt tuning. Changes to it create a new version.
- Metrics. Field-level precision, recall and F1 for extraction; recall@k and faithfulness for retrieval; task completion for workflows.
- Uncertainty. Bootstrap confidence intervals, and McNemar's test for paired comparisons between two systems.
Design choices and trade-offs
| Choice | Trade-off |
|---|---|
| LLM-as-judge for open-ended outputs | Scales cheaply; has position and verbosity biases, so it is calibrated against human labels (for example with Cohen's kappa) |
| Set size | Larger sets narrow the confidence intervals; labelling time is the constraint |
| Public benchmarks | Useful for a shortlist; may be contaminated by training data and rarely match the client's documents |
When not to use it
There is no system worth deploying that should skip this. A pilot can start with a small set and grow it from review-queue corrections.
How we evaluate it
The set itself is checked: label agreement between experts, coverage of document types against production volumes, and no overlap with any training data.
Sources
Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.