Profiling and verifying kernels

In one paragraph

A kernel is not finished when it is fast; it is finished when it is correct, fast on the real shapes, and measured honestly. Profiling finds where the time goes before any work starts, verification compares the kernel against a higher-precision reference with stated tolerances, and benchmarking follows rules that make the numbers repeatable.

How it works

• Profile. Nsight Systems for the timeline (gaps, launch overhead, copies); the PyTorch profiler to attribute time to model operations; Nsight Compute for one kernel in depth, including its roofline position.
• Verify. torch.testing.assert_close passes when |actual − expected| ≤ atol + rtol·|expected|. Defaults: bfloat16 rtol 1.6e-2, atol 1e-5; float16 rtol 1e-3, atol 1e-5; float32 rtol 1.3e-6, atol 1e-5.
• Test the edges. Shapes that are not multiples of the block size, masked and non-contiguous inputs, extreme values, and repeated runs for determinism.
• Benchmark. Warm up, repeat enough times to report a distribution, synchronise before reading timers, fix clocks where possible, and state shapes, dtypes, hardware and versions.

Design choices and trade-offs

• Reference precision. A float32 or float64 reference shows the kernel's error, not the reference's.
• Tolerances. Defaults suit most cases; reductions over long rows may need looser, justified tolerances, written down.

When not to use it

There is no case for shipping an unverified kernel.

How we evaluate it

Verification and benchmark scripts are kept with the kernel and run in CI on every change.

Sources

• PyTorch profiler recipe (opens in a new tab)
• torch.testing default tolerances (opens in a new tab)
• NVIDIA: roofline analysis in Nsight Compute (opens in a new tab)