# CUDA fundamentals for AI workloads

> The CUDA execution and memory model as it applies to AI kernels, including Hopper and Blackwell tensor-core features.

- Web page: https://www.alesanalytics.com/docs/cuda-fundamentals-for-ai-workloads
- Markdown: https://www.alesanalytics.com/docs/cuda-fundamentals-for-ai-workloads.md
- Last updated: 2026-11-08
- Publisher: Ales Analytics

Technical page. Warps, coalescing, shared memory and tensor cores. Published 2026-11-08. Last reviewed 2026-11-08. Reviewed by Ales Analytics engineering.

## In one paragraph

CUDA is NVIDIA's programming model for GPUs. Threads are grouped into warps of 32, warps into blocks, and blocks run on streaming multiprocessors that share fast on-chip memory. Writing fast kernels comes down to a few habits: read memory in coalesced patterns, use shared memory without bank conflicts, keep enough work in flight to hide latency, and feed the tensor cores.

## How it works

- **Coalescing.** A warp's 32 threads reading 32 consecutive addresses are served in a few wide transactions.
- **Shared memory banks.** 32 banks; several threads hitting the same bank at different addresses serialise. Padding tile rows avoids it.
- **Occupancy.** Resident warps per SM, limited by registers and shared memory. Higher occupancy hides latency, but using registers to keep data on-chip can matter more.
- **Tensor cores.** Units for small matrix multiplies at reduced precision. Hopper added the Tensor Memory Accelerator for asynchronous copies and warpgroup-level matrix instructions; Blackwell adds tensor memory for operands and accumulators.
- **Tools for writing them.** CUTLASS (C++ templates), the CuTe DSL (its layout abstractions from Python), and cuTile Python (tile-based Python).

![Figure 1. The GPU memory hierarchy.](https://www.alesanalytics.com/diagrams/gk-memory.svg)

## Design choices and trade-offs

| Choice                      | Trade-off                                                                    |
| --------------------------- | ---------------------------------------------------------------------------- |
| Raw CUDA vs CUTLASS or CuTe | Full control vs tested building blocks for matrix kernels                    |
| CUDA vs Triton              | More control over warps and new hardware features vs much faster development |

## When not to use it

When a Triton kernel or `torch.compile` reaches the target, writing CUDA by hand adds cost without benefit.

## How we evaluate it

Nsight Compute metrics for memory throughput, achieved occupancy and tensor-core utilisation, compared with the roofline bound for the kernel.

## Sources

- [CUDA C++ Best Practices Guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html)
- [CUTLASS CuTe DSL](https://docs.nvidia.com/cutlass/media/docs/pythonDSL/overview.html), [cuTile Python](https://developer.nvidia.com/blog/simplify-gpu-programming-with-nvidia-cuda-tile-in-python)

## Contact

solutions@alesanalytics.com, or https://www.alesanalytics.com/contact.

- Every page as Markdown: https://www.alesanalytics.com/llms.txt