Quantisation for inference

Fewer bits for weights, activations and cache

Published: 8 November 2026
Last reviewed: 8 November 2026
Re-checked every: 90 days
Reviewed by: Ales Analytics engineering

In one paragraph

Quantisation stores weights, activations or the attention cache in fewer bits. It cuts memory and bandwidth, which directly speeds up decode, and on hardware with matching tensor cores it can speed up the arithmetic too. The cost is some loss of accuracy, which varies by model, method and task, so every quantised model is checked against its full-precision baseline on the client's own test set.

How it works

Approach	What changes
Weight-only (AWQ, GPTQ)	Weights to around 4 bits; activations stay 16-bit
INT8, FP8	Weights and activations; FP8 is native on Hopper-class and newer GPUs
NVFP4, MXFP4	4-bit floating point with per-block scales (blocks of 16 with FP8 scales, or 32 with power-of-two scales)
KV-cache quantisation	The attention cache, often to FP8, doubling cache capacity relative to 16-bit

Design choices and trade-offs

Which tensors. Weight-only methods are easiest to apply; activation quantisation needs calibration and hardware support.
Calibration data. Methods such as AWQ and GPTQ use sample inputs; samples resembling the client's work give better results than generic text.
Where errors appear. Averages hide damage to numbers, rare fields and long documents.

When not to use it

When the model fits comfortably at 16-bit and latency targets are met, quantisation adds evaluation work for little gain.

How we evaluate it

Same frozen test set, same prompts, full precision against each quantised variant, with per-field and per-document-type results and the memory and latency each variant achieves.

Sources

Lin et al. (2023), AWQ (opens in a new tab); Frantar et al. (2022), GPTQ (opens in a new tab)
NVIDIA: introducing NVFP4 (opens in a new tab)

Should you have any questions regarding this page, please contact us at solutions@alesanalytics.com.