Why kernels matter

In one paragraph

A kernel is a function that runs on the GPU. A model's speed is the sum of its kernels' speeds, and each kernel is limited either by arithmetic or by memory bandwidth, depending on how many operations it does per byte it moves. Knowing which limit applies says what can make it faster: for memory-bound work, moving fewer bytes, usually by fusing operations; for compute-bound work, using tensor cores and lower precision well.

How it works

• Arithmetic intensity is operations per byte moved. The GPU's ratio of peak compute to bandwidth is the dividing line.
• The roofline plots attainable performance against intensity: a sloped line where bandwidth limits, a flat ceiling where compute limits.
• In a language model, elementwise operations, softmax, normalisation and small-batch decode are memory-bound; large matrix multiplies in prefill are compute-bound.
• Fusion combines several memory-bound operations into one kernel so intermediate results stay on-chip.

Design choices and trade-offs

• Library or custom. Vendor libraries and compiler output cover most needs. A custom kernel is a maintenance cost justified only by a measured bottleneck.
• Where the time goes. A kernel that is 2% of runtime cannot save more than 2%, however fast it becomes.

When not to use it

When a profile shows time spread evenly across library kernels, or dominated by data loading or the network, kernel work will not help.

How we evaluate it

Profile first, estimate the possible gain from the roofline, and only then decide whether a kernel is worth writing.

Sources

• NVIDIA: GPU performance background for deep learning (opens in a new tab)
• NVIDIA: roofline analysis in Nsight Compute (opens in a new tab)
• GPU kernel engineering, which works through decode intensity and fusion