Table of contents
In this article
Technical page · Models
Preference tuning and reinforcement learning with verifiable rewards
DPO, GRPO and distillation
- Published
- 8 November 2026
- Last reviewed
- 8 November 2026
- Re-checked every
- 90 days
- Reviewed by
- Ales Analytics engineering
In one paragraph
Preference tuning teaches a model to prefer better answers over worse ones, using pairs of outputs ranked by people. Reinforcement learning with verifiable rewards teaches it to pass checks that can be computed, such as producing valid JSON or matching a labelled value. Both come after supervised fine-tuning on the adaptation ladder, and both depend on the quality of the signal: a reward that can be gamed will be.
How it works
- DPO. Direct preference optimisation trains on (prompt, preferred, rejected) triples with a classification-style loss that raises the preferred answer's likelihood relative to a frozen reference model. β controls how far the model may move from the reference. Variants such as IPO, KTO, ORPO and SimPO change the loss or drop the reference.
- GRPO. Group relative policy optimisation samples several answers per prompt, scores each with a reward function, and uses each answer's reward relative to its group (minus the group mean, divided by the group standard deviation) as its advantage. No separate value network is needed.
- Verifiable rewards. Schema validity, exact field match, unit tests passing, arithmetic checking out.
- Distillation. A smaller student learns from a larger teacher's outputs, either complete answers (sequence-level) or the teacher's scoring of the student's own attempts (on-policy).
Design choices and trade-offs
| Choice | Trade-off |
|---|---|
| Human preferences vs computable rewards | Preferences capture quality that cannot be computed; rewards are cheap and consistent but narrow |
| Synthetic data | Cheap and plentiful; risks narrowing outputs (model collapse) and copying the teacher's errors |
| Hosted teacher | Often strongest; check the provider's terms, which may restrict training on outputs |
When not to use it
When supervised fine-tuning already meets the acceptance tests, or when no reliable preference or reward signal exists.
How we evaluate it
On the frozen test set, plus inspection of sampled outputs during training to catch reward hacking, such as valid but empty objects. Synthetic data is always mixed with real examples and checked against the frozen set.
Sources
- Rafailov et al. (2023), Direct Preference Optimization (opens in a new tab)
- Shao et al. (2024), DeepSeekMath (GRPO) (opens in a new tab)
- TRL GRPO trainer (opens in a new tab)
- Model adaptation and evaluation, which works through the DPO loss and a GRPO reward
Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.