Sizing a self-hosted deployment

In one paragraph

Sizing starts from memory. A deployment must hold the model's weights, the attention cache for every request in flight at its full context, and some headroom. From there, latency targets and expected load decide how many GPUs and which kind.

How it works

weight memory       = parameters × bytes per parameter
KV cache per token  = 2 × layers × KV heads × head dimension × bytes per value
total KV cache      = KV cache per token × context length × concurrent sequences
requests in flight  ≈ arrival rate × time per request        (Little's law)

Illustrative example. This explains a technique; it does not describe any client system. The model shape is hypothetical.

A hypothetical 8-billion-parameter model with 32 layers, 8 KV heads and head dimension 128, in 16-bit, needs 128 KiB of cache per token. Sixteen sequences at 8,192 tokens is 16 GiB of cache, plus about 16 GB of weights. With an FP8 cache, the cache halves to 8 GiB.

If the model does not fit on one GPU: tensor parallelism splits layers across GPUs in a server, pipeline parallelism places layers on different GPUs, expert parallelism places experts of a mixture-of-experts model, and data parallelism adds replicas.

Design choices and trade-offs

• Context length. Allowing longer context than the work needs wastes cache that could serve more users.
• Headroom. Typically 10 to 20% of GPU memory for activations, the runtime and fragmentation.
• Peak or average load. Sizing for peak wastes capacity; sizing for average queues at peak. Queueing at peak is often acceptable for back-office work and not for interactive use.

When not to use it

When a hosted API is the chosen route, sizing is the provider's problem; budget and rate limits replace it.

How we evaluate it

The sizing is written down with its assumptions, then checked by a load test on the target hardware before anything is bought.

Sources

• Inference performance, which works through this example and the bandwidth limit on decode
• Explainers: Tokens and inference, Owning or renting the hardware