Model adaptation and evaluation

Choosing a model

The first question is not which model is best, but which models are allowed. Three filters come before any accuracy test.

• Where it can run. An open-weight model is a file that can be served on the client's own GPUs. A hosted frontier model is reached over an API. Many systems use both, behind one gateway, with a written rule for which tasks may go outside.
• The licence. Open-weight families differ. Some are released under permissive licences such as Apache 2.0 or MIT; examples at the time of writing include Qwen3.5, Gemma 4, Mistral 3, DeepSeek-V4 and gpt-oss. Others use a custom licence, such as the Llama 4 Community License, which adds conditions including an attribution requirement, an acceptable-use policy and a separate licence for very large services. Hosted models have their own terms, and several restrict using their outputs to train other models, which matters if a hosted model is meant to generate training data. Check the current licence file; this is not legal advice.
• The shape. A dense model uses all its parameters for every token. A mixture-of-experts (MoE) model routes each token through a few experts, so its compute per token follows the active parameters while its memory follows the total. A model with 120 billion total and 5 billion active parameters is cheap to run per token but still needs memory for all 120 billion weights.

Only then does accuracy enter, and it is measured on the client's own test set, not on a public leaderboard.

The adaptation ladder

Each rung costs more than the one below it, so each is climbed only when evaluation shows the rung below is not enough.

1. 1Prompt and context engineering. Clear instructions, a fixed output schema, and a handful of examples from the client's own work.
2. 2Retrieval. If the model lacks knowledge (a price list, a policy, last quarter's figures), retrieval supplies it at question time. Retrieval adds knowledge; it is also the only rung where the knowledge can be updated the same day.
3. 3Supervised fine-tuning. If the model has the knowledge but behaves wrongly (wrong format, wrong tone, misses a field type it sees thousands of times a month), fine-tuning on labelled examples changes behaviour.
4. 4Preference tuning or reinforcement learning. If the gap is about choosing between acceptable and better outputs, or about reliably passing a check that can be computed, preference tuning or reinforcement learning with verifiable rewards is the next step.

The evaluation set is built before any training starts. Without it there is no way to know whether a rung helped.

Parameter-efficient fine-tuning

Full fine-tuning updates every weight and needs optimiser state for all of them. Parameter-efficient methods train a small number of extra parameters and leave the base model frozen.

• LoRA adds a low-rank update to selected weight matrices (see the worked example).
• QLoRA keeps the frozen base model in 4-bit NF4 precision, with double quantisation of the quantisation constants and paged optimisers to absorb memory spikes, so a large model can be adapted on a single GPU.
• DoRA splits each weight into a magnitude and a direction and applies the low-rank update to the direction only, which narrows the gap to full fine-tuning on some tasks.
• Merging or hot-swapping. A trained adapter can be merged into the base weights for the simplest serving, or kept separate so that one base model serves several adapters, one per task or per team, switched per request.
• Data hygiene. Training examples are deduplicated, stripped of personal data that is not needed for the task, and split so that a held-out set never touches training. Near-duplicates across the split are the most common way a test score ends up flattering.
• Tooling. Hugging Face TRL and PEFT, and Axolotl, are common options.

Research has found that LoRA tends to learn less of the new task than full fine-tuning but also forgets less of what the base model already knew, which is usually the right trade for a business task layered on a general model.

Preference tuning, reinforcement learning and distillation

• DPO (direct preference optimisation) learns from pairs of a preferred and a rejected answer without training a separate reward model. Successors such as IPO, KTO, ORPO and SimPO change the loss or remove the reference model; the choice is made by testing, not by publication date.
• GRPO with verifiable rewards. Where correctness can be computed (the output parses against the schema, a field exactly matches the label, generated code passes its tests), reinforcement learning can optimise directly against that check. GRPO compares several sampled answers to the same prompt with each other, so no separate value network is needed.
• Distillation. A larger teacher model generates outputs that a smaller student learns to imitate: sequence-level distillation trains on the teacher's complete answers, and on-policy distillation has the teacher score or correct the student's own attempts. The student is cheaper to serve.
• Synthetic data risks. Training repeatedly on model-generated data can narrow a model's output distribution until rare but important cases disappear, an effect described as model collapse. A student also copies its teacher's errors. Synthetic data is mixed with real examples, filtered, and checked against the frozen test set.

Evaluation built from your own data

• A frozen test set. Drawn from the client's own documents and questions, labelled by the client's own experts, and never used for training or prompt tuning. It is versioned; when it changes, scores before and after the change are not compared directly.
• Extraction. Field-level precision (of the values extracted, how many were right), recall (of the values present, how many were extracted) and F1, broken down by document type and field. One overall accuracy number hides the field that matters.
• Retrieval. Recall@k (did the right passage appear in the top k), and faithfulness of the answer to the retrieved passages.
• LLM-as-judge. A model can grade open-ended answers at scale, but judges have known biases: they favour the answer shown first (position bias) and longer answers (verbosity bias). A judge is calibrated against human labels before it is trusted (see the worked example).
• Contamination. Public benchmarks may have leaked into a model's training data. Private test sets drawn from the client's own documents avoid most of that risk, which is one more reason to build them.
• Tools. lm-evaluation-harness, Inspect and Ragas are common options, alongside plain scripts for task-specific metrics.

Evaluation in CI and production

• Regression thresholds. Every change to a prompt, model, adapter or retrieval setting runs the test set in CI. A drop beyond the agreed threshold on any tracked metric blocks the release.
• Shadow and canary releases. A new model first runs in shadow, receiving real traffic and logging its outputs without serving them. It then serves a small share of traffic before taking over.
• Drift signals. Changes in input mix (new document layouts, a new supplier), in confidence distributions and in review-queue volume are watched, because they usually show up before accuracy visibly drops.
• Sampled human review. A fixed sample of outputs is reviewed every week, and disagreements are added to the test set.
• Rollback. The previous model, adapter and prompt stay deployable, and switching back is a configuration change.

Governance mapping

Controls are mapped to the frameworks a client is likely to be asked about. The language is "aligned with", never "compliant".

See also the explainers on the EU AI Act, ISO/IEC 42001 and the NIST AI RMF.

Worked examples

LoRA in one line

Illustrative example. This explains a technique; it does not describe any client system.

A frozen weight matrix W of shape d × k is adapted as

W′ = W + (α / r) · B · A        B is d × r,  A is r × k,  r ≪ min(d, k)

Only A and B are trained. The trainable parameters fall from d·k to r·(d + k). For a 4096 × 4096 projection at r = 16, that is from 16,777,216 to 131,072, or about 0.8%. B is initialised to zero, so training starts from exactly the base model.

Variations: rsLoRA scales the update by α / √r instead of α / r, which keeps training stable at higher ranks. QLoRA stores W in 4-bit NF4 with double quantisation and uses paged optimisers. DoRA writes W as a magnitude vector times a unit-norm direction and applies the low-rank update to the direction only.

Preference tuning without a reward model

Illustrative example. This explains a technique; it does not describe any client system.

DPO minimises

L = −log σ( β · [ log πθ(y_w | x) − log π_ref(y_w | x)
                − log πθ(y_l | x) + log π_ref(y_l | x) ] )

where y_w is the preferred answer, y_l the rejected one, πθ the model being trained and π_ref a frozen copy of where it started. In effect this is a logistic classification loss on preference pairs: it raises the likelihood of the preferred answer relative to the rejected one, measured against the reference. β controls how far the model may move from the reference; a larger β keeps it closer.

Reinforcement learning with verifiable rewards

Illustrative example. This explains a technique; it does not describe any client system.

In GRPO, each prompt gets G sampled answers with rewards r₁ … r_G. Each answer's advantage is

A_i = (r_i − mean(r₁…r_G)) / std(r₁…r_G)

so answers better than their siblings are reinforced and worse ones discouraged, with no separate value network. A KL penalty keeps the policy near the reference model.

For an extraction task, a verifiable reward might be:

reward = 1.0 if output parses against the schema else 0.0
       + 1.0 × (fields exactly matching the label / fields in the label)

The reward is only as good as the check. A reward that can be satisfied by an empty but valid object will be satisfied that way, so rewards are inspected on sampled outputs during training.

Comparing two models fairly

Illustrative example. This explains a technique; it does not describe any client system.

Two candidate models are run on the same frozen test set. Field-level precision, recall and F1 are reported per document type, each with a bootstrap confidence interval (resample the documents with replacement, recompute, take the 2.5th and 97.5th percentiles).

For a paired right-or-wrong comparison on the same items, McNemar's test looks only at the items where the models disagree. Suppose, hypothetically, model A is right and B wrong on b = 31 fields, and B right and A wrong on c = 14:

χ² = (|b − c| − 1)² / (b + c) = (17 − 1)² / 45 ≈ 5.69

With one degree of freedom the 5% critical value is 3.84, so in this illustration the difference is unlikely to be noise. The fields both models get right, or both get wrong, say nothing about which is better and are ignored by the test.

Checking an LLM judge

Illustrative example. This explains a technique; it does not describe any client system.

1. 1Have the judge and two or more of the client's experts score the same sample of answers.
2. 2Measure agreement with Cohen's kappa, which corrects for agreement expected by chance. Compare judge-to-expert agreement with expert-to-expert agreement; a judge cannot be expected to agree with experts more than they agree with each other.
3. 3Swap the order of answers in pairwise comparisons. If the verdict flips, the judge has position bias on that item.
4. 4Pair a short correct answer with a long correct answer. If the judge consistently prefers the long one, it has verbosity bias.
5. 5Keep people on any category of output where judge and experts disagree.

Sources

• Model licences: Qwen3.5 (opens in a new tab), Gemma 4 (opens in a new tab), Mistral 3 (opens in a new tab), DeepSeek-V4 (opens in a new tab), gpt-oss (opens in a new tab), Llama 4 (opens in a new tab)
• Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models (opens in a new tab)
• Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs (opens in a new tab)
• Liu et al. (2024), DoRA: Weight-Decomposed Low-Rank Adaptation (opens in a new tab)
• Biderman et al. (2024), LoRA Learns Less and Forgets Less (opens in a new tab)
• Rafailov et al. (2023), Direct Preference Optimization (opens in a new tab)
• Shao et al. (2024), DeepSeekMath, introducing GRPO (opens in a new tab)
• Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (opens in a new tab)
• Hugging Face TRL v1 (opens in a new tab), TRL GRPO trainer (opens in a new tab), PEFT LoRA guide (opens in a new tab), Axolotl (opens in a new tab)
• lm-evaluation-harness (opens in a new tab), Inspect (opens in a new tab), Ragas faithfulness (opens in a new tab)
• OWASP Top 10 for LLM Applications (opens in a new tab), NIST AI RMF (opens in a new tab)
• White & Case: the EU AI Omnibus enters into force (opens in a new tab)