# Quantisation for inference

> Weight-only, FP8, NVFP4 and MXFP4 and KV-cache quantisation, and how each quantised model is checked against its full-precision baseline.

- Web page: https://www.alesanalytics.com/docs/quantisation-for-inference
- Markdown: https://www.alesanalytics.com/docs/quantisation-for-inference.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. Fewer bits for weights, activations and cache. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

Quantisation stores weights, activations or the attention cache in fewer bits. It cuts memory and bandwidth, which directly speeds up decode, and on hardware with matching tensor cores it can speed up the arithmetic too. The cost is some loss of accuracy, which varies by model, method and task, so every quantised model is checked against its full-precision baseline on the client's own test set.

## How it works

| Approach                | What changes                                                                                              |
| ----------------------- | --------------------------------------------------------------------------------------------------------- |
| Weight-only (AWQ, GPTQ) | Weights to around 4 bits; activations stay 16-bit                                                         |
| INT8, FP8               | Weights and activations; FP8 is native on Hopper-class and newer GPUs                                     |
| NVFP4, MXFP4            | 4-bit floating point with per-block scales (blocks of 16 with FP8 scales, or 32 with power-of-two scales) |
| KV-cache quantisation   | The attention cache, often to FP8, doubling cache capacity relative to 16-bit                             |

## Design choices and trade-offs

- **Which tensors.** Weight-only methods are easiest to apply; activation quantisation needs calibration and hardware support.
- **Calibration data.** Methods such as AWQ and GPTQ use sample inputs; samples resembling the client's work give better results than generic text.
- **Where errors appear.** Averages hide damage to numbers, rare fields and long documents.

## When not to use it

When the model fits comfortably at 16-bit and latency targets are met, quantisation adds evaluation work for little gain.

## How we evaluate it

Same frozen test set, same prompts, full precision against each quantised variant, with per-field and per-document-type results and the memory and latency each variant achieves.

## Sources

- Lin et al. (2023), [AWQ](https://arxiv.org/abs/2306.00978); Frantar et al. (2022), [GPTQ](https://arxiv.org/abs/2210.17323)
- [NVIDIA: introducing NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt