# Preference tuning and reinforcement learning with verifiable rewards

> How DPO and GRPO work, what verifiable rewards are, when distillation helps, and the risks of synthetic data and reward hacking.

- Web page: https://www.alesanalytics.com/docs/preference-tuning-and-rlvr
- Markdown: https://www.alesanalytics.com/docs/preference-tuning-and-rlvr.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. DPO, GRPO and distillation. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Preference tuning teaches a model to prefer better answers over worse ones, using pairs of outputs ranked by people. Reinforcement learning with verifiable rewards teaches it to pass checks that can be computed, such as producing valid JSON or matching a labelled value. Both come after supervised fine-tuning on the adaptation ladder, and both depend on the quality of the signal: a reward that can be gamed will be.

## How it works

- **DPO.** Direct preference optimisation trains on (prompt, preferred, rejected) triples with a classification-style loss that raises the preferred answer's likelihood relative to a frozen reference model. β controls how far the model may move from the reference. Variants such as IPO, KTO, ORPO and SimPO change the loss or drop the reference.
- **GRPO.** Group relative policy optimisation samples several answers per prompt, scores each with a reward function, and uses each answer's reward relative to its group (minus the group mean, divided by the group standard deviation) as its advantage. No separate value network is needed.
- **Verifiable rewards.** Schema validity, exact field match, unit tests passing, arithmetic checking out.
- **Distillation.** A smaller student learns from a larger teacher's outputs, either complete answers (sequence-level) or the teacher's scoring of the student's own attempts (on-policy).

## Design choices and trade-offs

| Choice                                  | Trade-off                                                                                        |
| --------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Human preferences vs computable rewards | Preferences capture quality that cannot be computed; rewards are cheap and consistent but narrow |
| Synthetic data                          | Cheap and plentiful; risks narrowing outputs (model collapse) and copying the teacher's errors   |
| Hosted teacher                          | Often strongest; check the provider's terms, which may restrict training on outputs              |

## When not to use it

When supervised fine-tuning already meets the acceptance tests, or when no reliable preference or reward signal exists.

## How we evaluate it

On the frozen test set, plus inspection of sampled outputs during training to catch reward hacking, such as valid but empty objects. Synthetic data is always mixed with real examples and checked against the frozen set.

## Sources

- Rafailov et al. (2023), [Direct Preference Optimization](https://arxiv.org/abs/2305.18290)
- Shao et al. (2024), [DeepSeekMath (GRPO)](https://arxiv.org/abs/2402.03300)
- [TRL GRPO trainer](https://huggingface.co/docs/trl/main/en/grpo_trainer)
- [Model adaptation and evaluation](https://www.alesanalytics.com/engineering/model-adaptation), which works through the DPO loss and a GRPO reward

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt