9 GPU Concepts Every AI Engineer Should Know

Frameworks like PyTorch are easy to use because they hide away all the intricacies that occur at the GPU level.

But companies and research labs today are looking for engineers who can look under the hood of PyTorch, explain where their compute budget is being spent, debug GPUs, and make LLMs more performant.

Here are 9 concepts explained simply that will help you better understand GPUs.

1, 2, 3: discuss GPU computation hardware
4, 5, 6: discuss GPU memory
7, 8: discuss how programs are executed on GPUs
9: discusses how multiple GPUs are connected together

Let's begin!

1. Streaming Multiprocessor

Streaming Multiprocessor (SM) is the core computational unit in NVIDIA GPUs.

It is similar to the cores of a CPU, with one major difference: it can execute parallel instructions and specific operations, such as matrix multiplication, much faster.

While a CPU core is optimized for high execution speed of a single thread, an SM on a GPU is optimized for high throughput across many threads (running a large number of threads in parallel).

A Streaming Multiprocessor (SM) is called so because:

It works with streams of data (data inputs requiring similar computation that are continuously fed to it)
It contains multiple processing units that can perform computations on a data stream in parallel

The main components of an SM are:

CUDA cores (INT32, FP32, FP64): For basic arithmetic calculations
Tensor Cores: For matrix multiplication calculations
Warp schedulers and Dispatch units: For choosing which thread groups (warps) to run next
Load/ Store (LD/ST) units: For moving data between memory and registers
Registers/ Register files: For storing each thread's data temporarily
L1 cache: For temporarily storing data to be used across the components of an SM
Shared memory: Partition of the same SRAM as the L1 cache that can be used to store data. While L1 caches data automatically, shared memory can be used programmatically.

The following is an architectural diagram of the SM for an NVIDIA H100 GPU. A single H100 SXM has 132 such SMs.

Although the above-mentioned components are most common in GPUs, you can see many more components in the H100 GPU, such as:

SFUs (Special Function Units): For performing exponential, logarithmic, and trigonometric functions.
Texture (Tex) units: For processing surface texture data efficiently
Tensor Memory Accelerator: For efficiently moving multi-dimensional data between global memory and shared memory
L0 and L1 instruction cache: Small and fast cache memory that stores frequently used instructions close to the compute cores

2. CUDA core

A CUDA (Compute Unified Device Architecture) core handles a single simple arithmetic operation, such as integer and floating-point calculations, per clock cycle, in an SM.

Different CUDA cores handle different types of numerical precision, such as INT32, FP32, and FP64, as seen in the architectural diagram above.

An H100 SXM GPU has:

128 FP32 CUDA cores per SM
16,896 FP32 CUDA cores per GPU

A CUDA core is much simpler than a CPU core. While a CPU core can perform complex operations, a CUDA core's job is to work as a simple calculator for basic arithmetic. It is the massive number of these cores performing calculations in parallel that makes a GPU so powerful.

Note that the term 'CUDA core' refers to hardware and is not directly related to the CUDA software ecosystem.

3. Tensor core

A Tensor core is a specialized unit in an SM that can perform fast matrix multiply-and-accumulate operations.

These operations make up the bulk of the workload in deep learning and are handled by the Tensor cores, leaving non-matrix operations (such as activation functions, normalization, and element-wise operations) to the CUDA cores.

NVIDIA first introduced the Tensor Core in the Volta GPU architecture in 2017, which massively accelerated neural-network training workflows.

Tensor Cores are also specialized for different numeric precisions, such as FP16, BF16, TF32, and FP64. The H100 GPU has specialized tensor cores for low-precision formats such as FP8, while the Blackwell generation of GPUs includes tensor cores for FP4 (specifically, NVFP4).

You will often find discussions of TFLOPS in GPU specs, which refer to the performance of the GPU's Tensor cores and CUDA cores.

4. High Bandwidth Memory

High Bandwidth Memory (HBM) is a GPU's main memory. It is also called global memory or VRAM. This is the memory that holds the model weights, activations, and KV cache.

HBM is mounted alongside the GPU die and works with smaller memory components (L2, L1, and register files) that are etched directly on the die (on-chip memory).

It is a type of Dynamic random-access memory (DRAM), which offers larger capacity and is cheaper than on-chip memory, a type of Static random-access memory (SRAM), which is extremely fast but much smaller.

5. On-chip memory

On-chip memory includes the memory components that are etched directly on the die and therefore called "on-chip".

These are a type of Static random-access memory (SRAM), which is extremely fast for reading and writing data, but much smaller than the HBM.

Arranged in ascending order of speed of access and descending order of capacity, these are:

L2 memory/ cache (shared across all SMs on a GPU)
L1 memory/ cache (shared across components of an SM)
Registers

In the H100 SM architecture diagram shown above, you can see the instruction and data caches labeled separately based on what they store. Also, 'Register files' means an array or collection of registers rather than a single one.

Algorithms like FlashAttention speed up computation by taking advantage of this memory hierarchy. FlashAttention splits and performs attention calculations in small blocks called Tiles that fit in on-chip memory, rather than repeatedly reading intermediate data from HBM.

6. Memory bandwidth

Memory bandwidth is the rate at which data moves between HBM and the SMs. It is usually reported in GB/s or TB/s.

The peak memory bandwidth for some popular NVIDIA GPUs is as follows:

L40S: 864 GB/s
A100 80GB SXM: 2 TB/s
H100 SXM: 3.35 TB/s
B100: 8 TB/s

For LLM inference, each decoding step loads all model weights from HBM to the compute cores (Tensor/CUDA cores) to generate a single token per sequence. This results in a memory bandwidth bottleneck because moving data is much slower than the rate at which the SMs can compute.

This is described as the Memory wall, where processor speed has improved much faster than memory speed. This makes most of LLM inference, specifically Decode, memory-bound rather than compute-bound.

7. Kernel

A kernel is a function that runs in parallel across many GPU threads, where each thread executes the same operation on different data. This is called the Single Instruction, Multiple Threads (SIMT) execution model.

A PyTorch function may launch one or more kernels depending on the operations required.

If you've heard of it, Kernel fusion is a popular technique that combines multiple GPU kernels into one, so that intermediate results stay in registers or shared memory rather than being written to and read back from global memory. FlashAttention is an example of a fused kernel for attention-related computations.

8. Thread, Warp, Block & Grid

CUDA (Compute Unified Device Architecture) is NVIDIA's software ecosystem for GPU programming. Programming execution with CUDA on a GPU is organized into multiple levels that are important to understand.

A Thread is the most basic unit of execution on a GPU. Each thread has its own private set of registers.

A group of 32 threads is called a Warp. All threads in a warp execute the same instruction in parallel on different data on an SM. This makes a warp the true unit of execution on a GPU.

Threads are grouped into Blocks that run on a single SM and share its shared memory. A block contains multiple threads that are automatically divided into warps for execution.

A Grid is a group of blocks that is launched with a kernel. Blocks in a grid are independent, and a GPU can schedule them onto any available SMs. This execution hierarchy makes a kernel scalable across SMs.

NVIDIA also introduced Thread Block Clusters in its Hopper architecture, in which multiple blocks are grouped to run together on neighboring SMs and efficiently share data using Distributed Shared Memory (DSM). Block clusters lie between blocks and grids in the CUDA execution hierarchy.

It is important to note that in CUDA, the programmer can define the number of threads, as well as the grid and block dimensions. However, a warp is not created directly by the programmer. Instead, a GPU automatically groups threads within each block into warps of 32 threads for execution.

9. PCIe, NVLink & NVSwitch

GPUs are rarely used in isolation but are connected to other GPUs to form massive data centers. They are arranged hierarchically as follows:

Server or Node (4 to 8 GPUs)
Server rack (10s to 100s GPUs)
Cluster or Pod (100s to 1000s GPUs)
Data center (10,000 to 100,000 GPUs)

Various interconnects are used for inter-GPU communication. The three important ones that we discuss are:

PCIe
NVLink
NVLink with NVSwitch

PCIe is the standard bus that connects hardware components, such as GPUs, to the CPU and to each other.

The 5th-generation PCIe offers 128 GB/s and the 6th-generation PCIe offers 256 GB/s of bidirectional GPU-to-GPU bandwidth.

Compare this to NVIDIA's proprietary GPU-to-GPU interconnect, NVLink, which is a massive bandwidth upgrade over PCIe. The bidirectional bandwidth between two GPUs provided by different generations of NVLink is as follows:

900 GB/s on H100 (NVLink 4)
1.8 TB/s on B200 (NVLink 5)
3.6 TB/s on Rubin (NVLink 6)

There's an issue, though: NVLink's total bandwidth is shared among the GPUs it connects. This means that if an H100 is connected to three other H100 GPUs via NVLink, its 900 GB/s bandwidth is split across the connections.

NVSwitch is another proprietary system from NVIDIA that provides a fix for this. It is a high-bandwidth, low-latency fabric that connects multiple GPUs within a single system, allowing them to share the full NVLink bandwidth on each connection.