# Why kernels matter

> Why GPU work is limited by compute or by bandwidth, how to read a roofline, and when kernel fusion helps.

- Web page: https://www.alesanalytics.com/docs/why-kernels-matter
- Markdown: https://www.alesanalytics.com/docs/why-kernels-matter.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. Arithmetic intensity, rooflines and fusion. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

A kernel is a function that runs on the GPU. A model's speed is the sum of its kernels' speeds, and each kernel is limited either by arithmetic or by memory bandwidth, depending on how many operations it does per byte it moves. Knowing which limit applies says what can make it faster: for memory-bound work, moving fewer bytes, usually by fusing operations; for compute-bound work, using tensor cores and lower precision well.

## How it works

- **Arithmetic intensity** is operations per byte moved. The GPU's ratio of peak compute to bandwidth is the dividing line.
- **The roofline** plots attainable performance against intensity: a sloped line where bandwidth limits, a flat ceiling where compute limits.
- **In a language model,** elementwise operations, softmax, normalisation and small-batch decode are memory-bound; large matrix multiplies in prefill are compute-bound.
- **Fusion** combines several memory-bound operations into one kernel so intermediate results stay on-chip.

![Figure 1. Roofline for an H100 SXM, from NVIDIA's published dense 16-bit figures. The x-axis doubles at each step.](https://www.alesanalytics.com/diagrams/gk-roofline.svg)

## Design choices and trade-offs

- **Library or custom.** Vendor libraries and compiler output cover most needs. A custom kernel is a maintenance cost justified only by a measured bottleneck.
- **Where the time goes.** A kernel that is 2% of runtime cannot save more than 2%, however fast it becomes.

## When not to use it

When a profile shows time spread evenly across library kernels, or dominated by data loading or the network, kernel work will not help.

## How we evaluate it

Profile first, estimate the possible gain from the roofline, and only then decide whether a kernel is worth writing.

## Sources

- [NVIDIA: GPU performance background for deep learning](https://docs.nvidia.com/deeplearning/performance/dl-performance-gpu-background/index.html)
- [NVIDIA: roofline analysis in Nsight Compute](https://developer.nvidia.com/blog/accelerating-hpc-applications-with-nsight-compute-roofline-analysis)
- [GPU kernel engineering](https://www.alesanalytics.com/engineering/gpu-kernels), which works through decode intensity and fusion

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt