Tuning and routing

In one paragraph

Tuning and routing are two ways of making an AI system faster, cheaper to run, and easier to check, without changing what it does for the person using it. Tuning adjusts a model using your own documents and examples of good answers, and is scored before and after to confirm it actually helped. Routing sends each question to the model best suited to it: a small, fast model for routine work, and a larger one only for the questions that genuinely need it.

Why it matters

Most of the questions people ask at work are routine and do not need a large, expensive model to answer well. Sending every question to the biggest available model regardless of how simple it is wastes capacity and makes the whole system slower and harder to audit than it needs to be. Tuning a smaller model on your own material, and routing questions to the right size of model for the job, keeps a system quick, keeps it easier to review, and keeps the larger model available for the cases that actually need its full capability.

How it works

• Tuning takes an existing model and adjusts it using a set of your own documents and examples of the answers you want, rather than building a model from nothing.
• A tuned model is scored against the same questions before and after the adjustment, so you can see whether the change actually improved anything rather than assuming it did.
• Routing sits in front of the models and decides, question by question, which one should handle it: routine and well understood questions go to a small, fast model, and only the genuinely difficult cases go to a larger one.
• The scoring set used to judge tuning is the same kind of fixed, reviewed set of questions used to check the system generally, which is what turns "did tuning help" into an answer you can check rather than a guess.

What it looks like in practice

Most of what an assistant gets asked is a routine lookup. A small number of questions are genuinely hard. Tuned on the team's own past answers, a small model handles the routine lookups quickly and accurately, scored against a set of real examples before and after the tuning to confirm it improved. Routing sends the harder, less common questions on to a larger model instead, so the system as a whole stays fast for most people most of the time, while still handling the difficult cases properly.

How this connects to our work

Sending each question to the right sized model is the substance of AI cost and model optimisation, and it is scored using the same evaluation sets that check the system's quality more generally.