# Evaluation built from your own data

> How to build, label and freeze an evaluation set from your own documents, which metrics to report, and how to calibrate an LLM judge.

- Web page: https://www.alesanalytics.com/docs/evaluation-from-your-own-data
- Markdown: https://www.alesanalytics.com/docs/evaluation-from-your-own-data.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. A frozen test set labelled by your own experts. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

An evaluation set built from the client's own data is a frozen collection of real inputs (documents, questions, cases) with correct outputs labelled by the client's own experts. It is the only reliable way to know whether a system works on the work it was built for, and every change to the system is scored against it.

## How it works

1. **Sample.** Draw real inputs covering the mix of types, layouts, sources and difficulty the system will see, including awkward cases.
2. **Label.** The client's experts label the correct output. Two labellers on a subset show how much people themselves disagree.
3. **Freeze and version.** The set is never used for training or prompt tuning. Changes to it create a new version.
4. **Metrics.** Field-level precision, recall and F1 for extraction; recall@k and faithfulness for retrieval; task completion for workflows.
5. **Uncertainty.** Bootstrap confidence intervals, and McNemar's test for paired comparisons between two systems.

## Design choices and trade-offs

| Choice                              | Trade-off                                                                                                                    |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| LLM-as-judge for open-ended outputs | Scales cheaply; has position and verbosity biases, so it is calibrated against human labels (for example with Cohen's kappa) |
| Set size                            | Larger sets narrow the confidence intervals; labelling time is the constraint                                                |
| Public benchmarks                   | Useful for a shortlist; may be contaminated by training data and rarely match the client's documents                         |

## When not to use it

There is no system worth deploying that should skip this. A pilot can start with a small set and grow it from review-queue corrections.

## How we evaluate it

The set itself is checked: label agreement between experts, coverage of document types against production volumes, and no overlap with any training data.

## Sources

- Zheng et al. (2023), [Judging LLM-as-a-Judge](https://arxiv.org/abs/2306.05685)
- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), [Inspect](https://inspect.aisi.org.uk/), [Ragas faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/)
- Explainer: [Evaluation sets](https://www.alesanalytics.com/explainers/evaluation-sets)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt