Tokens and inference

In one paragraph

A token is the unit a language model reads and writes in: roughly three-quarters of an English word, so a short paragraph might be a few dozen tokens and a long document several thousand. Inference is the act of the model producing an answer, one token at a time, in response to a question. Together, tokens and inference are what sizing an AI system actually measures: how many tokens a piece of work involves, and how quickly the hardware can produce them, decides how much a system can handle at once.

Why it matters

An AI system that works well in a small trial can behave quite differently once real documents and real numbers of people are involved. Longer documents mean more tokens to read before an answer can even begin. More people asking questions at the same time mean more of those answers being produced in parallel. Both push harder on the same piece of hardware. Understanding tokens and inference is what lets you size a system properly before it is running, rather than discovering the limits of your hardware once people are relying on it.

How it works

• A token is a small piece of text, close to three-quarters of a word on average in English, which is how a language model breaks language down to process it.
• Inference is what happens when the model turns a question, broken into tokens, into an answer, also produced token by token.
• How fast a system can produce tokens, its throughput, depends on the size of the model, the length of the documents involved, and how many questions are being answered at the same time.
• Longer documents take longer to read before an answer starts, and more concurrent questions divide the same hardware capacity between more people at once.
• Sizing a deployment means measuring these factors together, against your real documents and your real number of users, rather than guessing.

What it looks like in practice

A pilot runs with a handful of people asking short questions. Throughput looks generous, and answers come back quickly. Once the same assistant is opened up to a much larger group, some of whom paste in long documents rather than short questions, the same hardware is now producing many more tokens at once, and answers slow down. Sizing for the larger, more realistic pattern of use in advance is what avoids that surprise.

How this connects to our work

Measuring tokens and throughput on your real documents and your real number of users is how we size a private AI setup before anything is bought or built, so the system you end up with matches the load it actually needs to carry rather than the load it was tried out on.