CUDA fundamentals for AI workloads

In one paragraph

CUDA is NVIDIA's programming model for GPUs. Threads are grouped into warps of 32, warps into blocks, and blocks run on streaming multiprocessors that share fast on-chip memory. Writing fast kernels comes down to a few habits: read memory in coalesced patterns, use shared memory without bank conflicts, keep enough work in flight to hide latency, and feed the tensor cores.

How it works

• Coalescing. A warp's 32 threads reading 32 consecutive addresses are served in a few wide transactions.
• Shared memory banks. 32 banks; several threads hitting the same bank at different addresses serialise. Padding tile rows avoids it.
• Occupancy. Resident warps per SM, limited by registers and shared memory. Higher occupancy hides latency, but using registers to keep data on-chip can matter more.
• Tensor cores. Units for small matrix multiplies at reduced precision. Hopper added the Tensor Memory Accelerator for asynchronous copies and warpgroup-level matrix instructions; Blackwell adds tensor memory for operands and accumulators.
• Tools for writing them. CUTLASS (C++ templates), the CuTe DSL (its layout abstractions from Python), and cuTile Python (tile-based Python).

Design choices and trade-offs

When not to use it

When a Triton kernel or torch.compile reaches the target, writing CUDA by hand adds cost without benefit.

How we evaluate it

Nsight Compute metrics for memory throughput, achieved occupancy and tensor-core utilisation, compared with the roofline bound for the kernel.

Sources

• CUDA C++ Best Practices Guide (opens in a new tab)
• CUTLASS CuTe DSL (opens in a new tab), cuTile Python (opens in a new tab)