Skip to content

Chapter 4: GPUs & AI Accelerators

Modern AI, graphics and science run on chips built to do millions of sums at once: GPUs, TPUs, NPUs, FPGAs and a wave of new AI accelerators. This chapter explains why these chips beat ordinary CPUs at parallel work, how a GPU is organised inside and how it draws graphics, and who makes the key products as of late 2026, from NVIDIA Blackwell and Rubin to AMD Instinct, Google TPUs and laptop NPUs. You will also see how a chip is designed from RTL to tape-out, and learn the number formats, compute maths and networking that turn thousands of accelerators into one AI supercomputer.

4.1 Why Accelerators? CPUs vs GPUs and Specialised Chips

A CPU is a clever all-rounder that finishes one task at a time very quickly, while an accelerator is a chip built to do one kind of job, such as huge piles of multiplications, much faster and with less energy. Graphics, scientific simulation and AI all involve doing the same simple sum on millions of numbers at once, so a chip with thousands of small workers beats a chip with a few brilliant ones. Accelerators exist because ordinary chips stopped getting faster for free, so designers now win speed by specialising.

CPU vs GPU vs AI ASIC

CPU GPU AI ASIC (e.g. TPU)
Compute units 8-192 big cores Thousands of simple lanes Large matrix units
Optimised for Latency, branchy code Throughput, parallel maths One workload family
Memory DDR5, big caches HBM or GDDR, TB/s HBM plus large on-chip SRAM
Flexibility Runs anything Parallel programs Mostly neural networks
Programming C, Java, Python... CUDA, ROCm, Triton XLA, vendor SDKs
AI work per watt Lowest High Highest

Moving right trades flexibility for throughput and energy efficiency on parallel maths.

Think of it like…

A CPU is a few expert chefs who can cook any dish quickly; a GPU is a factory line of hundreds of helpers who can each only chop vegetables, but together they chop a mountain in minutes.

Real-world example

Leo, a physics student, simulates 10 million particles. His 8-core laptop CPU handles about 8 particles at a time and needs 40 minutes per run. His university's GPU runs thousands of particle updates in parallel and finishes in about 90 seconds. But when Leo parses a messy config file with lots of if-statements, the CPU is faster, because that job cannot be split into thousands of identical pieces.

How it works

  1. Ask whether the work is the same simple operation repeated on lots of data (parallel) or a long chain of decisions (serial).
  2. Parallel, maths-heavy work (graphics, matrices, AI) goes to a GPU or AI accelerator; branchy, serial work stays on the CPU.
  3. The CPU still runs the operating system, prepares the data and launches work on the accelerator.
  4. Check whether the job is limited by maths speed or by how fast memory can feed data (the roofline idea).
  5. Measure the real speed-up, remembering Amdahl's law: the serial part limits the total gain.

Who uses it

Machine-learning engineers, HPC scientists, game developers, data-centre architects and anyone choosing hardware for AI or simulation meets this trade-off.

Key takeaway

Memorise the one-liner: CPUs are optimised for latency (one task fast), GPUs for throughput (many tasks at once) — and most real systems use both together.

Going deeper

  • Specialisation wins by removing overhead: on a CPU most energy per instruction goes to fetching, decoding, scheduling and moving data, so dedicated datapaths can save tens of times the energy.
  • Roofline formula: attainable FLOPS = min(peak FLOPS, memory bandwidth × arithmetic intensity). Kernels left of the ridge point are memory-bound; those to the right are compute-bound.
  • Dennard scaling ended around 2005 and the power wall held clocks near 3-5 GHz, so performance growth moved to more cores, wider vectors and specialised accelerators.
  • Matrix multiplication dominates AI: an n × n matmul does 2n³ operations on 3n² values, so its arithmetic intensity grows with size, perfect for dense tensor units.
  • Accelerators still need a host CPU for the operating system, I/O and orchestration; system balance (PCIe or NVLink-C2C bandwidth, CPU memory, networking) often decides real speed.
  • TOPS and FLOPS figures are usually quoted at the lowest supported precision and sometimes with sparsity; normalise format (INT8, FP8, FP4) and dense versus sparse before comparing chips.

Common mistakes & misconceptions

  • Assuming a GPU speeds up any program: serial, branch-heavy or small workloads often run faster on the CPU.
  • Comparing peak TFLOPS across vendors or number formats directly: precision, sparsity and achievable utilisation differ widely.
  • Ignoring data transfer: copying data to and from the accelerator can erase the gain for small or one-off computations.

Try it: Work out a roofline: is a matrix multiply limited by maths or by memory?

# Roofline back-of-envelope in Python
peak_flops = 989e12      # e.g. H100 SXM dense BF16, FLOP/s
mem_bw     = 3.35e12     # H100 SXM HBM3, bytes/s
ridge = peak_flops / mem_bw
print(f'Ridge point: {ridge:.0f} FLOPs per byte')

n = 4096                 # square matrices, BF16 = 2 bytes
flops = 2 * n**3
bytes_moved = 3 * n * n * 2
ai = flops / bytes_moved
print(f'Matmul intensity: {ai:.0f} FLOPs/byte ->',
      'compute-bound' if ai > ridge else 'memory-bound')

Key chips, architectures, standards, tools & concepts

Name What it does Example
Domain-specific architecture (DSA)
Popularised by Hennessy & Patterson (Turing Lecture, 2018) · Concept · Current · Intermediate
A chip designed for one family of tasks, trading flexibility for big gains in speed and energy efficiency. Google's TPU drops features CPUs need for general code and spends the silicon on matrix-multiply units instead.
Roofline model
Williams, Waterman & Patterson, Communications of the ACM, 2009 · Concept · Current · Intermediate
A chart showing whether code is limited by peak compute or by memory bandwidth, based on its arithmetic intensity. Plotting a kernel on the roofline shows it sits under the memory slope, so faster memory, not more cores, will help.
Arithmetic intensity
Roofline model terminology · Metric · Current · Intermediate
Number of maths operations performed per byte moved from memory; high intensity means compute-bound, low means memory-bound. Large matrix multiplies reach hundreds of FLOPs per byte; adding two vectors manages well under one.
FLOPS and TOPS
Industry metrics · Metric · Current · Beginner
FLOPS counts floating-point operations per second; TOPS counts trillions of (usually integer) operations per second, often quoted for NPUs. A laptop NPU rated 50 TOPS at INT8 cannot be compared directly with a GPU's FP16 TFLOPS figure.
Heterogeneous computing
General architecture concept · Concept · Current · Beginner
Using different processor types (CPU, GPU, NPU, FPGA) together in one system, each doing the work it suits best. A phone's CPU runs the app, its GPU draws the interface and its NPU blurs the photo background.
Latency-optimised core (CPU style)
Intel, AMD, Arm, Apple and other CPU designs · Concept · Current · Beginner
A few large cores with out-of-order logic, branch predictors and big caches, spending silicon to finish one thread as fast as possible. A laptop CPU core runs a branchy JSON parser quickly because it predicts branches and reorders instructions on the fly.
Throughput-optimised core (GPU style)
NVIDIA, AMD, Intel, Apple and other GPU designs · Concept · Current · Beginner
Thousands of simple lanes that hide memory delays by switching between many threads, maximising total work rather than single-task speed. A GPU keeps tens of thousands of threads in flight, so while some wait for memory, others compute pixels or matrix tiles.
Systolic array
H. T. Kung and C. E. Leiserson (1978); used in Google TPUs · Architecture · Current · Intermediate
A grid of multiply-accumulate cells where data pulses from neighbour to neighbour, reusing each value many times without returning to memory. A 128 × 128 systolic array, as used in TPU matrix units, performs 16,384 multiply-accumulates every clock cycle.
Ridge point (machine balance)
Roofline model terminology · Metric · Current · Advanced
Peak FLOPS divided by memory bandwidth: the arithmetic intensity a kernel needs before compute, rather than memory, limits it. An H100 SXM with 989 dense BF16 TFLOPS and 3.35 TB/s has a ridge point near 295 FLOPs per byte.
Offload overhead
General heterogeneous-computing concept · Concept · Current · Intermediate
Time spent copying data over PCIe and launching work on an accelerator; for small jobs it can exceed the time the accelerator saves. Adding two 1,000-element arrays is slower on a GPU overall, because copying data and launching the kernel outlasts the sum itself.
Flexibility-efficiency spectrum
General computer-architecture concept · Concept · Current · Beginner
Chips range from flexible but less efficient (CPU) through GPU and FPGA to fixed-function but most efficient (ASIC). A start-up prototypes on GPUs, moves a stable video pipeline to an FPGA, then ships an ASIC once the design is final.

4.2 Inside a GPU: SMs, Warps, Tensor Cores and Memory

A GPU is built from dozens or hundreds of identical blocks (NVIDIA calls them Streaming Multiprocessors, AMD calls them Compute Units). Each block runs groups of threads in lockstep, all doing the same instruction on different data. Special units called tensor cores do small matrix multiplications in one step, which is exactly what AI needs, and a stack of very fast memory sits right next to the chip to keep them fed.

Fast code keeps reused data high in this stack and streams everything else from HBM.
GPU memory hierarchy (H100 example). Fast code keeps reused data high in this stack and streams everything else from HBM.

Think of it like…

A GPU is like a huge exam hall split into rooms (SMs); in each room an invigilator reads out one instruction and 32 students (a warp) all do that step on their own question at the same moment.

Real-world example

Aisha writes a kernel that adds two arrays of 1 million numbers. She launches 4,096 blocks of 256 threads; the GPU spreads the blocks across its SMs, and each SM runs its threads in warps of 32. Because neighbouring threads read neighbouring numbers, memory accesses are combined and the job takes well under a millisecond. When she later adds an if-statement that sends half of each warp down a different path, the warps must run both paths one after the other and her kernel slows noticeably.

How it works

  1. The CPU copies data to GPU memory and launches a kernel: a grid of thread blocks.
  2. The GPU scheduler hands blocks to SMs (NVIDIA) or CUs (AMD), which split them into warps of 32 or wavefronts of 64 threads.
  3. Each warp executes one instruction across all its threads (SIMT); while one warp waits for memory, the SM switches to another to hide the delay.
  4. Matrix-heavy instructions go to tensor cores (matrix cores on AMD); ray tracing goes to RT cores.
  5. Threads use registers first, then shared memory/L1, then L2 cache, then off-chip HBM or GDDR memory.
  6. Results are written back to GPU memory and copied to the CPU when needed.

Who uses it

CUDA/HIP developers, ML performance engineers, game-engine programmers and computer-architecture students need this mental model to write fast code.

Key takeaway

Learn the ladder registers → shared memory/L1 → L2 → HBM: each step down is larger but slower, and good GPU code keeps reused data as high up the ladder as possible.

Going deeper

  • An H100 SXM has 132 SMs, each with four warp schedulers, 128 FP32 lanes and four tensor cores, and can keep up to 64 warps (2,048 threads) resident per SM.
  • GPUs hide latency instead of avoiding it: when one warp waits hundreds of cycles for memory, the scheduler issues from another ready warp at no switching cost, since every warp's registers stay resident.
  • Tensor cores gained formats each generation: FP16 (Volta), TF32, BF16 and 2:4 sparsity (Ampere), FP8 (Hopper), then FP6 and FP4 with micro-scaling (Blackwell).
  • Hopper added thread-block clusters, distributed shared memory and the Tensor Memory Accelerator (TMA) for asynchronous bulk copies, letting data loading overlap tensor-core maths.
  • Occupancy is limited by registers per thread, shared memory per block and threads per block; Nsight Compute shows which resource is the limiter.
  • Single-thread speed is modest: GPU clocks run around 1.5-2.5 GHz, each warp issues in order, and there is no deep branch prediction or speculation like on CPUs.

Common mistakes & misconceptions

  • Equating CUDA cores with CPU cores: a CUDA core is one arithmetic lane, not an independent processor with its own instruction stream.
  • Launching too few threads: a kernel with a handful of blocks leaves most SMs idle; aim for many blocks per SM.
  • Ignoring access patterns: strided or uncoalesced memory access can make a kernel several times slower than its maths suggests.
  • Assuming more registers per thread always helps: heavy register use lowers occupancy and can spill to slow local memory.

Try it: Ask your NVIDIA GPU about its SM count, memory and clocks

nvidia-smi --query-gpu=name,memory.total,clocks.max.sm,utilization.gpu --format=csv
python -c "import torch;p=torch.cuda.get_device_properties(0);print(p.name,p.multi_processor_count,'SMs',p.total_memory//2**30,'GiB')"

Key chips, architectures, standards, tools & concepts

Name What it does Example
Streaming Multiprocessor (SM)
NVIDIA · Component · Current · Intermediate
NVIDIA's basic GPU building block, containing CUDA cores, tensor cores, registers, schedulers and shared memory/L1 cache. An H100 SXM has 132 SMs enabled; a laptop RTX GPU might have around 20-40.
Compute Unit (CU)
AMD (CDNA, RDNA); Intel uses Xe-cores · Component · Current · Intermediate
AMD's equivalent of an SM: a cluster of SIMD units, registers and local data share memory. An AMD Radeon RX 9070 XT has 64 CUs; an Instinct MI300X has 304.
Warp / wavefront
NVIDIA (warp, 32 threads); AMD (wavefront, 64 on CDNA, 32 or 64 on RDNA) · Concept · Current · Intermediate
A group of threads that the hardware schedules together and that execute the same instruction at the same time. A block of 256 threads on an NVIDIA GPU is run as 8 warps of 32 threads each.
SIMT (Single Instruction, Multiple Threads)
NVIDIA term (Tesla architecture, 2006) · Architecture · Current · Intermediate
Execution model where each thread has its own data and registers, but threads in a warp share one instruction stream. Code is written for one thread, yet the hardware runs it for 32 threads per instruction issued.
Warp divergence
GPU programming concept · Concept · Current · Advanced
When threads in one warp take different branches, the warp runs each path in turn, wasting lanes and slowing the kernel. An if-statement on odd versus even thread IDs can roughly halve that section's speed.
CUDA core
NVIDIA · Component · Current · Beginner
Marketing name for one scalar FP32/INT32 arithmetic lane inside an SM; GPUs have thousands of them. The GeForce RTX 5090 is listed with 21,760 CUDA cores.
Tensor Core
NVIDIA (first in Volta V100, 2017); AMD calls its versions Matrix Cores · Component · Current · Beginner
A unit that multiplies small matrices and adds the result in one operation, giving huge speed-ups for AI. Training in BF16 on tensor cores is many times faster than using the same GPU's ordinary FP32 lanes.
RT Core
NVIDIA (first in Turing, 2018); AMD Ray Accelerators · Component · Current · Beginner
Hardware that speeds up ray-tracing tests, checking which triangles a light ray hits, for realistic graphics. A game turns on ray-traced reflections; RT cores handle the ray-triangle tests while shader cores colour pixels.
GPU register file
All GPU vendors · Component · Current · Advanced
The fastest storage on the GPU, private to each thread; using too many registers per thread reduces how many warps fit. Each H100 SM has a 256 KB register file shared among all the threads it hosts.
Shared memory / L1 cache
NVIDIA (shared memory); AMD (LDS, Local Data Share) · Component · Current · Intermediate
Small, very fast on-chip memory that threads in a block can share, used to reuse tiles of data. A matrix-multiply kernel loads 64x64 tiles into shared memory so each value is read from HBM only once.
GPU L2 cache
All GPU vendors · Component · Current · Intermediate
Large on-chip cache shared by all of a GPU's SMs, catching repeated accesses before they reach HBM or GDDR memory. The H100 has a 50 MB L2 cache; AMD's MI300X adds a 256 MB Infinity Cache.
GDDR memory
JEDEC (GDDR6, GDDR7) · Component · Current · Beginner
Graphics DRAM soldered around the GPU on the board; cheaper than HBM with lower bandwidth, used in gaming and workstation cards. The RTX 5090 uses 32 GB of GDDR7 on a 512-bit bus.
Occupancy
NVIDIA / GPU programming term · Metric · Current · Advanced
The ratio of active warps on an SM to the maximum possible; higher occupancy helps hide memory delays. Cutting registers per thread from 128 to 64 let twice as many warps run, raising occupancy and speed.
GPU memory bandwidth
Vendor datasheets · Metric · Current · Beginner
Bytes per second the GPU can move between its HBM or GDDR memory and the chip; often the true limit for AI inference. An H200's roughly 4.8 TB/s lets it read its entire 141 GB memory about 34 times every second.
Memory coalescing
CUDA and HIP programming guides · Concept · Current · Intermediate
Combining neighbouring threads' memory accesses into a few wide transactions, achieved when consecutive threads read consecutive addresses. Threads 0-31 reading floats 0-31 need one 128-byte transaction; reading every 32nd float needs 32 separate ones.

4.3 NVIDIA GPUs: From V100 to Blackwell, Rubin and GeForce RTX 50

NVIDIA is the biggest maker of AI accelerators, and it releases a new data-centre GPU family roughly every year or two, each named after a scientist: Volta, Ampere, Hopper, Blackwell and now Rubin. Each generation adds more memory, faster links between GPUs and support for smaller, faster number formats. The same architectures also appear, scaled down, in GeForce gaming cards and RTX PRO workstation cards.

Roughly yearly launches, each adding memory, link bandwidth and a smaller number format.
NVIDIA data-centre GPU generations. Roughly yearly launches, each adding memory, link bandwidth and a smaller number format.

Think of it like…

NVIDIA's generations are like yearly smartphone models: the shape stays familiar, but each one has more memory, a faster chip and new tricks the previous model lacked.

Real-world example

A fictional AI start-up, Northwind Labs, trained its first model on rented A100s in 2022. In 2024 it moved to H100s, which use FP8 and finished a similar job in roughly a third of the time. In 2025 it reserved part of a GB200 NVL72 rack, where 72 Blackwell GPUs share one NVLink domain and act like one giant GPU. For 2027 it is evaluating Vera Rubin NVL72 cloud capacity, which only began shipping in late 2026.

How it works

  1. Volta V100 (2017) introduced tensor cores; Ampere A100 (2020) added TF32, BF16 and Multi-Instance GPU.
  2. Hopper H100 (2022) and H200 (2024) added FP8 and the Transformer Engine, with up to 141 GB of HBM3e.
  3. Blackwell B200/GB200 (2024-25) joined two dies in one package, added FP4 and the 72-GPU NVL72 rack.
  4. Blackwell Ultra B300/GB300 (2025) raised memory to 288 GB HBM3e per GPU.
  5. Vera Rubin (production shipments from August 2026) brings HBM4, NVLink 6 and the Vera CPU; Rubin CPX and Rubin Ultra follow.
  6. GeForce RTX 50 (Blackwell, 2025) and RTX PRO cards bring the same ideas to gamers and workstations.

Who uses it

AI researchers, cloud providers, HPC centres, gamers and anyone renting GPU instances meet these product names daily.

Key takeaway

Learn each generation with one headline feature: V100 = tensor cores, A100 = TF32/MIG, H100 = FP8, B200 = FP4 and NVL72, Rubin = HBM4 and NVLink 6.

Going deeper

  • Data-centre GPUs come as SXM modules on HGX baseboards, for full NVLink bandwidth and higher power, or as PCIe cards that fit standard servers with lower limits.
  • Power per GPU has climbed steeply: about 300 W for V100 SXM, up to 700 W for H100 SXM and roughly 1,000-1,400 W for Blackwell parts, pushing data centres to liquid cooling.
  • The rack is now the product: a GB200 NVL72 draws roughly 120 kW and ships as one liquid-cooled unit with NVLink switch trays, power shelves and cabling.
  • NVIDIA moves whole platforms each generation (GPU, Grace or Vera CPU, NVLink, ConnectX NICs, switches), so buyers plan around platform generations rather than single chips.
  • Software longevity is a selling point: CUDA, cuDNN, NCCL, TensorRT and NGC containers span generations, though new formats such as FP4 need recent drivers and libraries.
  • US export controls restrict the most powerful AI GPUs for some countries, which has led NVIDIA to offer cut-down variants for those markets.

Common mistakes & misconceptions

  • Comparing rack figures with single-GPU figures: check whether quoted petaFLOPS or bandwidth is per GPU, per board or per NVL72 rack.
  • Assuming GeForce gaming cards suit data centres: memory, NVLink, cooling, support and licensing terms differ from data-centre parts.
  • Reading roadmap dates as delivery dates: new parts reach the largest customers first and appear in public clouds months later.

Try it: Identify your NVIDIA GPU, driver and CUDA version

nvidia-smi
nvidia-smi -L                # list GPUs with UUIDs
nvidia-smi -q -d MEMORY,POWER | more

Key chips, architectures, standards, tools & concepts

Name What it does Example
Tesla V100 (Volta)
NVIDIA, 2017 · Accelerator · Legacy · Beginner
First GPU with tensor cores, with 16 or 32 GB HBM2; powered the Summit supercomputer. Many university clusters still run V100s for teaching, though they lack BF16 and FP8.
A100 (Ampere)
NVIDIA, 2020 · Accelerator · Current · Beginner
40 or 80 GB HBM data-centre GPU that introduced TF32, BF16 tensor cores and Multi-Instance GPU (MIG) partitioning. A cloud provider splits one A100 into seven MIG slices so seven students each get an isolated GPU.
H100 (Hopper)
NVIDIA, 2022 · Accelerator · Current · Beginner
80 GB HBM3 GPU adding FP8 tensor cores and the Transformer Engine; the workhorse of the 2023-24 AI boom. An H100 SXM offers about 3.35 TB/s of memory bandwidth and 900 GB/s NVLink 4 per GPU.
H200 (Hopper)
NVIDIA, 2024 · Accelerator · Current · Intermediate
Hopper GPU upgraded to 141 GB of HBM3e at about 4.8 TB/s, helping larger models fit and run faster. A 70-billion-parameter model in FP8 (about 70 GB) fits on one H200 with room for KV cache.
B200 (Blackwell)
NVIDIA, 2024-25 · Accelerator · Current · Intermediate
Two large dies joined into one GPU with about 208 billion transistors, up to 192 GB HBM3e, around 8 TB/s and FP4 support. An HGX B200 server board carries eight B200 GPUs linked by fifth-generation NVLink.
GB200 NVL72
NVIDIA, 2025 · System · Current · Intermediate
Liquid-cooled rack joining 72 Blackwell GPUs and 36 Grace CPUs in one NVLink domain that behaves like one huge GPU. Its NVLink switch spine gives about 130 TB/s of total GPU-to-GPU bandwidth inside a single rack.
B300 / GB300 NVL72 (Blackwell Ultra)
NVIDIA, shipping from second half of 2025 · Accelerator · Current · Intermediate
Upgraded Blackwell with 288 GB HBM3e per GPU and stronger FP4, aimed at reasoning-model inference. Cloud providers deployed GB300 NVL72 racks during 2025-26 for large reasoning-model serving.
Rubin GPU and Vera Rubin NVL72
NVIDIA; shown at CES 2026, production shipments from August 2026 · System · Emerging · Intermediate
Next generation: Rubin GPUs with 288 GB HBM4 and NVLink 6, paired with Vera Arm CPUs in a 72-GPU rack. NVIDIA quotes about 50 petaFLOPS of NVFP4 inference per Rubin GPU; initial shipments of Vera Rubin systems began in the third quarter of 2026.
Rubin CPX
NVIDIA, announced September 2025 · Accelerator · Emerging · Advanced
A Rubin-family GPU with 128 GB GDDR7, specialised for the prefill (context-reading) stage of very long-context inference. Planned NVL144 CPX racks pair Rubin CPX for reading million-token prompts with standard Rubin GPUs for generating answers.
Rubin Ultra
NVIDIA, announced for 2027 · Accelerator · Emerging · Advanced
Planned follow-up packing four GPU dies and HBM4E per package into a much denser 'Kyber' rack. NVIDIA roadmaps show Rubin Ultra after Vera Rubin, followed by a generation codenamed Feynman.
NVLink-C2C superchips (Grace and Vera)
NVIDIA (Arm-based) · Technology · Current · Intermediate
Pairing NVIDIA's Arm CPUs with its GPUs over a coherent chip-to-chip link (900 GB/s on Grace) so both share memory quickly. In GB200 each Grace CPU feeds two Blackwell GPUs; in Vera Rubin, Vera CPUs take that role.
DGX and HGX
NVIDIA · System · Current · Beginner
DGX is NVIDIA's complete AI server or SuperPOD; HGX is the GPU baseboard other server makers build around. A university buys HGX B200 servers from a server vendor instead of a DGX to save on cost.
GeForce RTX 5090
NVIDIA, January 2025 (Blackwell) · Accelerator · Current · Beginner
Flagship consumer card with 32 GB GDDR7, fifth-generation tensor cores and DLSS 4, popular with gamers and local-AI hobbyists. A student runs a quantised 30-billion-parameter model locally on one RTX 5090.
RTX PRO 6000 Blackwell
NVIDIA, 2025 · Accelerator · Current · Intermediate
Workstation and server card with 96 GB GDDR7 for 3D design, simulation and local AI development. An architecture firm renders a building walkthrough and fine-tunes a design-assistant model on the same card.
CUDA compute capability
NVIDIA · Concept · Current · Intermediate
Version number of a GPU's feature set that compilers target: 7.0 for V100, 8.0 for A100, 9.0 for H100, 10.0 for B200. Building with nvcc -arch=sm_90 produces code for H100-class GPUs; an older A100 cannot run that binary.

4.4 AMD Instinct and Radeon GPUs

AMD is NVIDIA's main GPU rival. Its Instinct accelerators (based on the CDNA architecture) power supercomputers and AI clouds and are known for carrying lots of memory, while its Radeon cards (RDNA architecture) target gamers and workstations. AMD builds its big chips from several smaller 'chiplets' stacked and joined in one package.

AMD's lever each generation: more HBM per GPU, then rack-scale systems.
AMD Instinct generations. AMD's lever each generation: more HBM per GPU, then rack-scale systems.

Think of it like…

If NVIDIA is the market-leading car brand, AMD is the strong challenger that often offers a bigger boot (more memory) for the money, and now sells whole buses (rack systems) too.

Real-world example

Marta runs inference for a fictional legal-search firm, Briefwise. Her 405-billion-parameter model in FP8 needs about 405 GB just for weights. Eight MI300X GPUs give 1.5 TB of HBM3 in one server, so the model fits with plenty of room for KV cache. When she upgrades to MI355X (288 GB each), the same server holds about 2.3 TB and can also run FP4.

How it works

  1. CDNA architecture powers Instinct data-centre GPUs; RDNA powers Radeon gaming cards.
  2. MI250X (2021) powered Frontier; MI300X/MI300A (2023) brought 3D-stacked chiplets and 128-192 GB of HBM3.
  3. MI325X (2024) and MI350X/MI355X (2025) raised memory to 256 and 288 GB HBM3E and added FP4/FP6.
  4. MI400 series and the 72-GPU Helios rack (shipping from late 2026) add HBM4 and rack-scale design.
  5. Software runs through ROCm and HIP; PyTorch and vLLM support AMD GPUs directly.

Who uses it

Supercomputer centres, cloud providers, AI labs looking for a second supplier, and gamers buying Radeon cards.

Key takeaway

Remember AMD's selling point in one phrase: 'more HBM per GPU' — and that El Capitan (MI300A) and Frontier (MI250X) are AMD-powered exascale machines.

Going deeper

  • MI300X stacks eight XCD compute chiplets on four I/O dies using 3D hybrid bonding, surrounded by eight HBM3 stacks, a showcase of chiplet design.
  • CDNA runs 64-thread wavefronts (RDNA supports 32 and 64), so code tuned for NVIDIA's 32-thread warps may need changes to reductions, shuffles and occupancy tuning.
  • ROCm now ships faster with broader support: PyTorch, JAX, vLLM and SGLang run on Instinct, though niche CUDA-only libraries can still need porting through HIP.
  • Memory capacity is AMD's main lever: more HBM per GPU means fewer GPUs to hold a large model and its KV cache, lowering inference cost.
  • Infinity Fabric links up to eight Instinct GPUs per server today; Helios moves AMD to a 72-GPU rack-scale domain, with open standards such as UALink and Ultra Ethernet on its roadmap.
  • Radeon and Instinct share chiplet and packaging know-how but differ in focus: RDNA keeps display, ray-tracing and media engines, while CDNA drops them for matrix cores and FP64.

Common mistakes & misconceptions

  • Assuming CUDA code runs on AMD unchanged: it must be ported to HIP or run through frameworks with ROCm support.
  • Assuming every Radeon card is officially supported by ROCm: check AMD's compatibility list for your GPU and operating system.
  • Comparing AMD and NVIDIA on peak FLOPS alone: memory capacity, software maturity and networking often decide real cost.

Try it: Inspect AMD GPUs on a Linux machine with ROCm installed

rocm-smi                          # temperatures, power, utilisation
rocm-smi --showmeminfo vram
rocminfo | grep -E 'Marketing Name|Compute Unit'
amd-smi list                      # newer replacement tool

Key chips, architectures, standards, tools & concepts

Name What it does Example
CDNA architecture
AMD · Architecture · Current · Intermediate
AMD's compute-focused GPU design for data centres, stripped of display hardware and built around matrix cores. CDNA 3 is used in MI300X; CDNA 4 in MI350/MI355X.
RDNA architecture
AMD · Architecture · Current · Beginner
AMD's graphics-focused GPU design for Radeon cards, PlayStation 5 and Xbox Series consoles. RDNA 4 (2025) improved ray tracing and added AI hardware for FSR 4 upscaling.
Instinct MI250X
AMD, 2021 (CDNA 2) · Accelerator · Legacy · Intermediate
Two-die accelerator with 128 GB HBM2e and strong FP64, built for supercomputers. Oak Ridge's Frontier, the first exascale system on TOP500 (2022), uses MI250X GPUs.
Instinct MI300X
AMD, December 2023 (CDNA 3) · Accelerator · Current · Beginner
Chiplet GPU with 192 GB HBM3 at about 5.3 TB/s, aimed at large-model AI inference and training. Cloud providers offer eight-GPU MI300X servers with 1.5 TB of combined HBM.
Instinct MI300A
AMD, 2023 · Accelerator · Current · Intermediate
APU combining 24 Zen 4 CPU cores and CDNA 3 GPU chiplets sharing 128 GB of unified HBM3. Lawrence Livermore's El Capitan supercomputer is built from MI300A APUs, so CPU and GPU never copy data between memories.
Instinct MI325X
AMD, late 2024 · Accelerator · Current · Intermediate
MI300-class GPU refreshed with 256 GB HBM3E at about 6 TB/s. A team serving long-context chat picks MI325X so each GPU holds a bigger KV cache.
Instinct MI350X / MI355X
AMD, June 2025 (CDNA 4, 3 nm) · Accelerator · Current · Intermediate
GPUs with 288 GB HBM3E, about 8 TB/s and new FP4/FP6 support; MI355X is the higher-power liquid-cooled version. AMD quotes roughly 9-10 petaFLOPS of FP4 compute for one MI355X.
Instinct MI400 series (MI455X, MI430X, MI440X)
AMD, announced 2025-26 · Accelerator · Emerging · Advanced
HBM4 generation: MI455X for AI racks, MI430X for FP64-heavy supercomputers, MI440X for eight-GPU enterprise servers. France's planned Alice Recoque exascale system is slated to use MI430X.
Helios rack
AMD, launched 2026 · System · Emerging · Intermediate
Double-wide liquid-cooled rack with 72 MI455X GPUs, EPYC 'Venice' CPUs and Pensando networking, rivalling NVIDIA's NVL72. AMD said Helios shipments begin around the end of Q3 2026, with OpenAI deployments from Q4 2026.
Instinct MI500 series
AMD, previewed for 2027 · Accelerator · Emerging · Advanced
Announced next generation after MI400, previewed by AMD for 2027. AMD's annual roadmap shows MI500 following MI400 so buyers can plan multi-year deployments.
Radeon RX 9070 XT
AMD, March 2025 (RDNA 4) · Accelerator · Current · Beginner
Mid-to-high-end gaming card with 64 compute units and 16 GB GDDR6. A gamer plays at 1440p with FSR 4 upscaling and also runs small AI models through ROCm on Linux.
Radeon AI PRO R9700
AMD, 2025 (RDNA 4) · Accelerator · Current · Intermediate
Workstation card with 32 GB GDDR6 aimed at local AI inference and development. A developer fits a quantised 30-billion-parameter model on one card for offline testing.
Ryzen AI Max (Strix Halo)
AMD, 2025 · Processor · Current · Intermediate
Laptop/mini-PC chip combining Zen 5 CPU, large RDNA 3.5 GPU and NPU with up to 128 GB unified memory. Hobbyists use Ryzen AI Max mini-PCs to run 70-billion-parameter models locally in shared memory.
AMD-OpenAI agreement
AMD and OpenAI, October 2025 · Concept · Current · Intermediate
Deal for OpenAI to deploy 6 gigawatts of AMD Instinct GPUs over several years, with warrants for up to 160 million AMD shares. The first gigawatt is planned on MI450-series GPUs from the second half of 2026, giving AMD a flagship AI customer.

4.5 Intel GPUs and Gaudi AI Accelerators

Intel, famous for CPUs, also makes graphics chips and AI accelerators, but its data-centre efforts have been bumpy. Its Gaudi chips are cost-focused AI accelerators, its big Ponte Vecchio GPU powered a supercomputer before being discontinued, and its Arc cards compete in budget gaming. New inference GPUs and a rack-scale system are on Intel's roadmap.

A mix of shipped products, a cancelled GPU and an inference-first restart.
Intel's accelerator journey. A mix of shipped products, a cancelled GPU and an inference-first restart.

Think of it like…

Intel's accelerator story is like a famous restaurant opening a new cuisine: some dishes did well, some were taken off the menu, and the next menu is still being written.

Real-world example

A fictional college, Harbourside Tech, builds a teaching cluster on a tight budget. It buys Arc Pro B60 cards with 24 GB each to teach PyTorch through Intel's XPU backend, and rents Gaudi 3 time for a research project because the quoted price per job is lower than the GPUs it compared. Students also use OpenVINO to run the trained model on laptop NPUs.

How it works

  1. Intel's Xe GPU architecture spans integrated graphics, Arc gaming cards and data-centre parts.
  2. Data Center GPU Max (Ponte Vecchio) powered the Aurora supercomputer but was discontinued.
  3. Gaudi (from the Habana Labs acquisition, 2019) offers AI accelerators with built-in Ethernet; Gaudi 3 launched in 2024.
  4. Falcon Shores was cut back to an internal test chip; Crescent Island (inference GPU) and Jaguar Shores (rack-scale) are next.
  5. Software uses oneAPI, SYCL, the PyTorch XPU backend and OpenVINO.

Who uses it

Budget-conscious AI teams, HPC centres with Intel systems, PC gamers and developers targeting Intel laptops.

Key takeaway

Know the code names: Ponte Vecchio (discontinued), Gaudi 3 (shipping), Crescent Island and Jaguar Shores (upcoming) — interviewers like candidates who track roadmaps.

Going deeper

  • Gaudi differs from GPUs: matrix engines plus programmable tensor processor cores, with RoCE Ethernet ports on the chip itself, so clusters scale out without separate network cards.
  • Intel's pitch is cost and openness: standard Ethernet, oneAPI and SYCL and OpenVINO rather than proprietary interconnects, aimed at enterprises not training frontier models.
  • Aurora at Argonne combines 63,744 Ponte Vecchio GPUs with 21,248 Xeon Max CPUs and passed one exaFLOPS on HPL in 2024.
  • Integrated Xe graphics ship in most Intel laptops, so Intel GPUs reach far more users than its discrete cards; XMX engines in Arc and Xe3 accelerate AI locally.
  • Arc Pro B-series cards and Crescent Island use cheaper GDDR6 or LPDDR5X instead of HBM, trading bandwidth for capacity and cost, a strategy aimed at inference rather than training.

Common mistakes & misconceptions

  • Assuming roadmap chips will ship as announced: Falcon Shores shows plans can be cancelled or turned into test vehicles.
  • Confusing Intel's NPU, integrated GPU and Gaudi: they target laptop AI features, client graphics and data-centre AI respectively.
  • Expecting CUDA-specific code to run on Intel hardware: use SYCL, oneAPI or framework backends such as PyTorch XPU.

Try it: List OpenCL devices (works for Intel, AMD and NVIDIA GPUs)

clinfo -l                     # short list of platforms and devices
clinfo | grep -E 'Device Name|Max compute units|Global memory size'
python -c "import torch;print(torch.xpu.is_available())"   # Intel GPU in PyTorch

Key chips, architectures, standards, tools & concepts

Name What it does Example
Xe architecture family
Intel · Architecture · Current · Intermediate
Intel's GPU design family, from Xe-LP integrated graphics to Xe-HPC data-centre parts and the newer Xe2 and Xe3 generations. Battlemage Arc cards use Xe2; Panther Lake laptop chips use Xe3 integrated graphics.
Data Center GPU Max (Ponte Vecchio)
Intel, 2023 · Accelerator · Legacy · Intermediate
Huge multi-tile HPC GPU with over 100 billion transistors; Intel has since wound the line down. Argonne's Aurora exascale supercomputer uses tens of thousands of these GPUs.
Gaudi 2
Intel (Habana Labs), 2022 · Accelerator · Legacy · Intermediate
AI training accelerator with 96 GB HBM2e and 24 built-in 100 GbE ports for scale-out. Some clouds offered Gaudi 2 instances as a cheaper alternative to A100s.
Gaudi 3
Intel, 2024 · Accelerator · Current · Intermediate
AI accelerator with 128 GB HBM2e and 24 integrated 200 GbE RoCE ports, sold on price-performance. A firm builds a Gaudi 3 cluster linked by standard Ethernet switches rather than InfiniBand.
Falcon Shores
Intel · Accelerator · Legacy · Advanced
Planned GPU merging Gaudi and Xe ideas; in early 2025 Intel said it would remain an internal test chip only. Analysts cite Falcon Shores as an example of a roadmap product that never reached customers.
Crescent Island
Intel, announced October 2025 · Accelerator · Emerging · Intermediate
Inference-focused data-centre GPU on the Xe3P architecture with 160 GB of LPDDR5X memory; customer sampling planned for late 2026. Intel pitches it for air-cooled servers running token-heavy inference cheaply.
Jaguar Shores
Intel, announced · System · Emerging · Advanced
Intel's planned rack-scale AI platform, successor to the Falcon Shores effort. Intel presents Jaguar Shores as its answer to NVL72-style racks, but launch timing is not yet firm.
Arc B580
Intel, December 2024 (Battlemage) · Accelerator · Current · Beginner
Budget gaming card with 12 GB GDDR6, XeSS upscaling and AI XMX matrix engines. A student builds a 1440p gaming PC around an Arc B580 to keep costs low.
Arc Pro B60
Intel, 2025 · Accelerator · Current · Intermediate
Workstation card with 24 GB GDDR6 marketed for multi-GPU AI inference boxes. A small lab packs four B60 cards into one workstation for 96 GB of total GPU memory.
XMX engines
Intel · Component · Current · Intermediate
Xe Matrix eXtensions: Intel's matrix-multiply units, equivalent to NVIDIA tensor cores. XeSS upscaling runs its neural network on the Arc card's XMX engines.
OpenVINO
Intel (open source) · Framework · Current · Intermediate
Toolkit that optimises trained models and runs them on Intel CPUs, GPUs and NPUs. A shop converts a camera model with OpenVINO so it runs on a laptop NPU without a discrete GPU.
Intel-NVIDIA collaboration
Intel and NVIDIA, announced September 2025 · Concept · Emerging · Intermediate
Partnership in which NVIDIA invested in Intel and the two plan x86 chips with RTX GPU chiplets and custom x86 CPUs for NVIDIA platforms. Future laptops may combine an Intel x86 CPU and NVIDIA RTX graphics in one package.

4.6 Programming GPUs: CUDA, ROCm, SYCL, Triton and Friends

To use a GPU you write small functions called kernels that run on thousands of threads at once, or, more often, you call libraries and frameworks that already contain fast kernels. NVIDIA's CUDA is the most widely used platform; AMD's ROCm, Khronos standards like OpenCL, SYCL and Vulkan, and newer tools like Triton offer alternatives. Most AI developers work in PyTorch or JAX and only drop down to kernels for the last bit of speed.

Most developers stay near the top; performance engineers work down into kernels and libraries.
The GPU software stack. Most developers stay near the top; performance engineers work down into kernels and libraries.

Think of it like…

Frameworks like PyTorch are a microwave meal (quick and good enough), libraries like cuBLAS are a professional kitchen's ready-made sauces, and hand-written CUDA kernels are cooking from scratch for the very best result.

Real-world example

Tom fine-tunes a model in PyTorch; one line, model.to('cuda'), moves it to the GPU and PyTorch calls cuBLAS and cuDNN underneath. Profiling with Nsight Systems shows a custom activation function wasting time, so he writes a 30-line Triton kernel that fuses three operations into one and saves 12% per step. His colleague runs the same PyTorch code unchanged on an AMD MI300X, because PyTorch's ROCm build maps the same calls to HIP and rocBLAS.

How it works

  1. Start in a framework (PyTorch, JAX) that already uses GPU libraries.
  2. Libraries like cuBLAS, cuDNN, rocBLAS and oneDNN provide tuned kernels for maths and neural-network layers.
  3. For custom operations, write kernels in Triton (Python-like) or CUDA/HIP (C++-like).
  4. Multi-GPU jobs use NCCL (NVIDIA) or RCCL (AMD) for collective communication.
  5. Profile with Nsight Systems/Compute or ROCm tools, then optimise memory access first.

Who uses it

ML engineers, HPC developers, graphics programmers and compiler engineers; CUDA skills appear in many AI-infrastructure job adverts.

Key takeaway

In interviews, explain why CUDA is a moat: nearly two decades of libraries, tools and community code — and know that HIP is designed to look almost identical to CUDA for easy porting.

Going deeper

  • CUDA code compiles to PTX, a virtual instruction set, and to SASS machine code for specific GPUs; shipping PTX lets the driver JIT-compile it for newer GPUs.
  • A sensible optimisation order: use libraries first, fuse operations to cut memory traffic, tile data into shared memory, then tune occupancy and instruction mix with Nsight Compute.
  • Streams and CUDA Graphs overlap independent kernels and copies and cut launch overhead, which matters when a model runs thousands of small kernels per step.
  • Portability layers include HIP (runs on AMD and NVIDIA), SYCL, Kokkos, OpenMP offload and Vulkan compute; AI frameworks hide most of this from everyday developers.
  • torch.compile captures Python models as graphs and generates fused Triton kernels, often giving speed-ups without any hand-written CUDA.
  • NCCL and RCCL pick ring or tree algorithms based on topology and message size, so correct detection of NVLink, PCIe and network layout is vital for multi-GPU speed.

Common mistakes & misconceptions

  • Timing kernels without synchronising: launches are asynchronous, so measure with CUDA events or cudaDeviceSynchronize, not just CPU timers.
  • Copying data between CPU and GPU every step: keep tensors resident on the GPU and transfer only what you need.
  • Hand-writing kernels before profiling: time often goes to data loading or a few library calls, not the code you suspect.

Try it: A minimal CUDA kernel that adds two vectors (save as add.cu, build with nvcc add.cu -o add)

#include <cstdio>
__global__ void add(const float* a, const float* b, float* c, int n) {
  int i = blockIdx.x * blockDim.x + threadIdx.x;   // global thread id
  if (i < n) c[i] = a[i] + b[i];
}
int main() {
  int n = 1 << 20; size_t sz = n * sizeof(float);
  float *a, *b, *c;
  cudaMallocManaged(&a, sz); cudaMallocManaged(&b, sz); cudaMallocManaged(&c, sz);
  for (int i = 0; i < n; i++) { a[i] = 1.0f; b[i] = 2.0f; }
  add<<<(n + 255) / 256, 256>>>(a, b, c, n);   // grid of 256-thread blocks
  cudaDeviceSynchronize();
  printf("c[0] = %f\n", c[0]);                  // prints 3.000000
  cudaFree(a); cudaFree(b); cudaFree(c);
}

Key chips, architectures, standards, tools & concepts

Name What it does Example
CUDA
NVIDIA, first released 2007 · Language · Current · Intermediate
NVIDIA's parallel computing platform: a C/C++ extension, compiler (nvcc), runtime and huge library ecosystem for its GPUs only. A kernel marked __global__ is launched with add<<<blocks, threads>>>(...) syntax from host code.
ROCm and HIP
AMD (open source) · Software · Current · Intermediate
AMD's GPU software stack; HIP is its CUDA-like C++ language, and hipify tools convert CUDA code automatically. A lab runs hipify-perl on its CUDA simulation and compiles it for MI300A with few manual fixes.
OpenCL
Khronos Group (OpenCL 3.0) · Standard · Current · Intermediate
Open, vendor-neutral standard for running kernels on GPUs, CPUs and FPGAs; widely supported but less used for AI today. A photo-editing app uses OpenCL so its filters accelerate on Intel, AMD and NVIDIA GPUs alike.
SYCL and oneAPI
Khronos Group (SYCL 2020); Intel oneAPI, now under the UXL Foundation · Language · Current · Advanced
Single-source modern C++ model for heterogeneous devices; oneAPI DPC++ and AdaptiveCpp are implementations. Aurora supercomputer codes use SYCL so they could also be compiled for other vendors' GPUs.
Triton
OpenAI (open source, 2021) · Language · Current · Advanced
Python-based language for writing fast GPU kernels in blocks, without managing threads and shared memory by hand. PyTorch's torch.compile generates Triton kernels automatically to fuse several operations together.
Metal and MLX
Apple · Framework · Current · Intermediate
Metal is Apple's graphics and compute API; MLX is Apple's open-source array framework for machine learning on Apple silicon. A student fine-tunes a small language model on a MacBook using MLX and its unified memory.
Vulkan compute
Khronos Group (2016) · Standard · Current · Advanced
Low-level cross-platform graphics API that also runs compute shaders on almost any GPU, including phones. llama.cpp's Vulkan backend lets one build run local AI on AMD, Intel and NVIDIA GPUs.
DirectX 12, DirectML and Windows ML
Microsoft · Framework · Current · Intermediate
Windows GPU APIs; DirectML runs neural networks on any DirectX 12 GPU, and Windows ML now picks CPU, GPU or NPU automatically. A Windows photo app ships one ONNX model and lets Windows ML run it on whichever chip is present.
WebGPU
W3C GPU for the Web Working Group · Standard · Current · Intermediate
Modern browser API giving web pages access to GPU graphics and compute, successor to WebGL. A website runs a small speech-recognition model in the browser on the visitor's own GPU.
cuBLAS and cuDNN
NVIDIA · Framework · Current · Intermediate
GPU libraries for dense linear algebra (cuBLAS) and deep-neural-network layers such as convolution and attention (cuDNN). PyTorch calls cuBLAS for every large matrix multiply, so users rarely write their own.
NCCL and RCCL
NVIDIA (NCCL); AMD (RCCL) · Framework · Current · Advanced
Libraries for fast multi-GPU collective operations such as all-reduce over NVLink, PCIe, InfiniBand or Ethernet. During data-parallel training, NCCL averages gradients across 64 GPUs after every step.
PyTorch
PyTorch Foundation (Linux Foundation); originally Meta · Framework · Current · Beginner
The most popular deep-learning framework; runs on NVIDIA, AMD, Intel and Apple GPUs through different backends. x = torch.randn(4096, 4096, device='cuda') creates a matrix directly in GPU memory.
JAX and XLA
Google (open source) · Framework · Current · Intermediate
JAX is a NumPy-like library that compiles code with the XLA compiler for GPUs and TPUs. Researchers write a model once in JAX and train it on Cloud TPUs or NVIDIA GPUs.
Nsight Systems and Nsight Compute
NVIDIA · Tool · Current · Advanced
Profilers showing a timeline of CPU and GPU activity (Systems) and detailed per-kernel metrics (Compute). Nsight Systems reveals the GPU idles 30% of the time waiting for the data loader.
Kernel, grid, block and thread
CUDA and HIP programming model (OpenCL: NDRange, work-group, work-item) · Concept · Current · Beginner
A kernel launches as a grid of blocks; threads in a block share fast memory and can synchronise, while blocks run independently. add<<<4096, 256>>> launches 4,096 blocks of 256 threads, about one million threads, each adding one pair of elements.

4.7 Graphics: The Rendering Pipeline, Ray Tracing and Displays

GPUs were invented to draw pictures: they turn 3D scenes made of triangles into millions of coloured pixels, many times every second. The steps — moving triangles into place, working out which pixels they cover, colouring each pixel and sending the finished frame to the screen — form the graphics pipeline. Modern GPUs add ray tracing for realistic light and AI upscaling to keep frame rates high.

Ray tracing plugs into this pipeline for selected effects; rasterisation still draws most of a game's image.
The real-time rendering pipeline. Ray tracing plugs into this pipeline for selected effects; rasterisation still draws most of a game's image.

Think of it like…

Rendering is like a stage crew rebuilding every frame of a film live: one team positions the scenery (vertices), another works out what each seat can see (rasterisation), and painters colour every visible spot (pixel shading) — over a hundred times a second.

Real-world example

Aisha's racing game targets 120 frames per second, so each frame must be finished in about 8.3 ms. The CPU sends draw calls for the cars, track and crowd; the GPU transforms millions of triangles, rasterises them and shades about 8.3 million pixels for a 4K screen. Ray-traced reflections are too slow at full resolution, so she renders at 1440p and lets AI upscaling rebuild a 4K image, while variable refresh rate keeps the monitor in step with each frame.

How it works

  1. The game engine on the CPU decides what might be visible and issues draw calls through an API such as DirectX 12, Vulkan or Metal.
  2. Vertex shaders transform each triangle's corners from 3D world positions into screen positions.
  3. The rasteriser works out which pixels each triangle covers and creates fragments for them.
  4. Pixel (fragment) shaders compute colours from textures, lights and materials, and the depth buffer keeps only the nearest surface.
  5. Ray-tracing units trace rays for reflections, shadows and lighting, and AI upscalers and denoisers fill in detail.
  6. The finished frame waits in a frame buffer until the display engine sends it over DisplayPort or HDMI at the next refresh.

Who uses it

Game and graphics programmers, technical artists, GPU driver and architecture engineers, visual-effects and visualisation studios, and simulation and digital-twin developers work with the graphics pipeline.

Key takeaway

Think in frame time, not just frames per second: 60 fps means 16.7 ms per frame and 120 fps means 8.3 ms, and the slowest stage — CPU, GPU or memory — sets the limit.

Going deeper

  • Shaders are small programs running on the same GPU cores used for compute; one 4K frame launches millions of pixel-shader threads, which is why GPUs were massively parallel long before AI arrived.
  • Ray-tracing hardware speeds up two jobs: walking a bounding volume hierarchy (BVH) of nested boxes around the scene, and testing rays against triangles; denoisers then clean up the few rays per pixel affordable in real time.
  • Path tracing follows light bouncing many times for film-quality lighting; offline renderers spend minutes to hours per frame, while games use it only with heavy upscaling and denoising.
  • AI upscalers such as NVIDIA DLSS 4, AMD FSR 4 and Intel XeSS 2 render fewer pixels and reconstruct the rest with neural networks on tensor or matrix units; frame generation inserts AI-predicted frames between rendered ones.
  • Generated frames improve smoothness but not responsiveness, because input is only sampled on rendered frames; latency reducers such as NVIDIA Reflex and AMD Anti-Lag shorten the render queue to compensate.
  • Mobile GPUs use tile-based rendering, shading the screen in small tiles held in on-chip memory to save DRAM bandwidth and power, an approach desktop GPUs have adopted only partly.
  • Fixed-function media engines encode and decode H.264, HEVC and AV1 video, so streaming a game or joining a video call barely loads the shader cores.

Common mistakes & misconceptions

  • Judging smoothness by average fps alone: stutter comes from occasional long frames, so check 1% lows or a frame-time graph.
  • Believing frame generation reduces input lag: it adds visual smoothness but slightly increases latency, so competitive players often turn it off.
  • Assuming ray tracing has replaced rasterisation: games are hybrids that rasterise the scene and ray-trace selected effects such as reflections, shadows and global lighting.

Try it: A minimal vertex and fragment shader pair (GLSL)

// vertex shader: place each corner on screen
#version 330 core
layout(location = 0) in vec3 pos;
uniform mat4 mvp;
void main() { gl_Position = mvp * vec4(pos, 1.0); }

// fragment shader: colour every covered pixel orange
#version 330 core
out vec4 colour;
void main() { colour = vec4(1.0, 0.5, 0.0, 1.0); }

Key chips, architectures, standards, tools & concepts

Name What it does Example
Graphics pipeline
Graphics APIs (Direct3D, Vulkan, OpenGL, Metal) · Concept · Current · Beginner
The fixed order of stages that turns 3D geometry into a 2D image: vertex processing, rasterisation, pixel shading and output. Every frame of a game flows through the pipeline, from a car's 3D model to coloured pixels on the screen.
Draw call
Graphics APIs · Concept · Current · Intermediate
A command from the CPU asking the GPU to render a batch of geometry with given shaders, textures and settings. A city scene issuing 20,000 draw calls may be CPU-bound; instancing draws a thousand identical trees in one call.
Vertex and pixel (fragment) shaders
Graphics APIs; HLSL, GLSL and Metal Shading Language · Concept · Current · Intermediate
Programmable GPU stages: vertex shaders position geometry, and pixel shaders compute the colour of every covered pixel. A water shader moves vertices up and down to make waves, then adds reflections and foam per pixel.
Rasterisation
General graphics technique · Concept · Current · Beginner
Converting triangles into the set of screen pixels they cover, producing fragments that are then shaded. A triangle covering a quarter of a 1080p screen produces about 500,000 fragments to shade.
Texture mapping and texture units
GPU hardware · Concept · Current · Intermediate
Wrapping images onto surfaces; texture units fetch and filter texels quickly, and mipmaps reduce shimmering on distant surfaces. A brick wall is a flat rectangle plus a brick texture and a normal map that fakes the depth of the mortar.
Depth buffer (Z-buffer)
General graphics technique · Concept · Current · Intermediate
A per-pixel record of the distance to the nearest surface drawn so far, so hidden surfaces are discarded whatever the drawing order. A car drawn after the building behind it still correctly hides that building's pixels.
Frame buffer and double buffering
General graphics technique · Concept · Current · Beginner
Memory holding the finished image; drawing into a hidden back buffer while showing the front buffer prevents half-drawn frames. Swapping buffers only at the display's refresh (V-Sync) prevents tearing, at the cost of some extra latency.
Ray tracing and BVH
NVIDIA RT cores, AMD and Intel ray accelerators; DXR and Vulkan ray tracing · Technology · Current · Intermediate
Simulating light rays to find what they hit, using a bounding volume hierarchy to skip most of the scene quickly. Ray-traced reflections in a puddle can show objects behind the camera, which screen-space tricks cannot.
Path tracing
Film renderers; some real-time games · Technology · Current · Advanced
A physically based method that follows many random light bounces per pixel to produce realistic global illumination. Animated films render each frame with path tracers on render farms, often taking hours per frame.
AI upscaling and frame generation (DLSS, FSR, XeSS)
NVIDIA; AMD; Intel · Technology · Current · Intermediate
Neural networks rebuild high-resolution frames from lower-resolution renders and can insert generated frames between real ones. DLSS 4 multi-frame generation on GeForce RTX 50 cards can show up to three generated frames for each rendered frame.
Frame rate, frame time and refresh rate
General · Metric · Current · Beginner
Frame rate counts frames rendered per second, frame time is each frame's duration, and refresh rate is how often the display updates. A 144 Hz monitor refreshes every 6.9 ms; a game averaging 100 fps with occasional 30 ms frames still feels jerky.
Variable refresh rate (G-SYNC, FreeSync)
NVIDIA; AMD; VESA Adaptive-Sync; HDMI VRR · Technology · Current · Beginner
The display waits for each new frame instead of refreshing on a fixed clock, removing tearing without V-Sync's extra lag. With FreeSync, a game dipping from 120 to 90 fps stays smooth because the monitor follows the GPU's pace.
Display panels (LCD, OLED, mini-LED)
Display industry · Technology · Current · Beginner
LCDs filter a backlight, mini-LED backlights add thousands of dimming zones, and OLED pixels emit their own light for true blacks. A designer picks an OLED monitor for its contrast but hides static toolbars to reduce burn-in risk.
Video encode and decode engines (H.264, HEVC, AV1)
NVIDIA NVENC/NVDEC; AMD VCN; Intel Quick Sync; Apple media engine · Component · Current · Intermediate
Fixed-function blocks that compress and decompress video far more efficiently than shader cores or the CPU. A streamer encodes 1440p AV1 on the GPU's encoder while the game keeps almost all of its frame rate.
Game engines (Unreal Engine, Unity, Godot)
Epic Games; Unity Technologies; Godot Foundation · Software · Current · Beginner
Toolkits that handle rendering, physics, audio and scripting, so developers build games and simulations without writing a renderer. A car maker builds an interactive digital twin of its showroom in Unreal Engine 5.

4.8 Cloud Giants' Own AI Chips: TPU, Trainium, Maia and MTIA

The biggest cloud and internet companies now design their own AI chips instead of relying only on GPUs. Google's TPUs, Amazon's Trainium and Inferentia, Microsoft's Maia and Meta's MTIA are tuned for each company's own workloads and data centres. Outside customers can usually only rent them through that company's cloud, not buy them.

Each cloud builds chips for its own workloads, mostly with design partners and TSMC manufacturing.
Hyperscalers' own AI chips. Each cloud builds chips for its own workloads, mostly with design partners and TSMC manufacturing.

Think of it like…

It is like a large supermarket chain launching its own-brand products: they are made for its own shelves, often cheaper per unit, and you can only get them in that shop.

Real-world example

A fictional video platform, StreamNest, compares costs for serving a recommendation model. Its own benchmark shows that serving on AWS Inferentia2 instances would cost about 35% less per million predictions than its rented GPUs, and a Google Cloud TPU v5e test gives a similar saving. The catch is porting: the team must compile its PyTorch model through the AWS Neuron SDK or use JAX/PyTorch-XLA for TPUs, which costs two engineer-weeks. They decide the saving pays back within a few months.

How it works

  1. A cloud company spots a huge, predictable workload (search, ads ranking, chat assistants).
  2. It designs a custom chip (often with a partner such as Broadcom or Marvell) and has it made by TSMC.
  3. Chips are linked into pods or UltraServers using custom high-speed interconnects.
  4. A software layer (XLA, Neuron SDK, Maia SDK) lets PyTorch or JAX code run on the chip.
  5. The company uses the chip internally and rents it to customers as cloud instances.

Who uses it

Cloud architects, ML platform engineers and finance teams comparing cost per token or per training run.

Key takeaway

Remember which cloud owns which chip: Google = TPU, AWS = Trainium/Inferentia, Microsoft = Maia, Meta = MTIA (internal only).

Going deeper

  • Custom chips cut cost per token on a company's own predictable workloads by dropping features hyperscalers do not need, such as graphics, strong FP64 and general programmability.
  • TPU pods connect chips with inter-chip interconnect links in a 3D torus, and Google uses optical circuit switches to reconfigure slices and route around failed hardware.
  • Software is the real barrier: AWS Neuron, Google's XLA and JAX stack and Microsoft's Maia SDK must cover every operator a model uses, so porting effort often decides adoption.
  • Most custom accelerators use HBM from the same three suppliers and TSMC advanced packaging, so they compete with GPUs for the same scarce supply.
  • The 2025-26 trend is splitting chips by phase: inference-focused parts such as Ironwood, Maia 200 and TPU 8i emphasise memory capacity and bandwidth over raw training FLOPS.

Common mistakes & misconceptions

  • Assuming you can buy TPUs or Trainium for your own data centre: they are generally available only as cloud services.
  • Choosing a chip from datasheets alone: hourly price, availability, software support and porting time dominate total cost.
  • Expecting every PyTorch model to run unchanged: unsupported operators may need rewrites or fall back to slower paths.

Try it: List available Cloud TPU types in a zone (read-only, needs the gcloud CLI and a project)

gcloud compute tpus accelerator-types list --zone=us-central1-a
aws ec2 describe-instance-types --filters Name=instance-type,Values=trn2* \
  --query 'InstanceTypes[].InstanceType' --region us-east-1

Key chips, architectures, standards, tools & concepts

Name What it does Example
TPU (Tensor Processing Unit)
Google, in use internally since 2015 · Accelerator · Current · Beginner
Google's AI chip built around large systolic arrays of matrix-multiply units, linked into pods with optical circuit switches. Google trains and serves its Gemini models largely on TPU pods.
Cloud TPU v5e and v5p
Google, 2023 · Accelerator · Current · Intermediate
v5e targets cost-efficient training and inference; v5p is the high-performance version scaling to pods of 8,960 chips. A start-up fine-tunes on a v5e slice to keep its monthly bill low.
Trillium (TPU v6e)
Google, generally available late 2024 · Accelerator · Current · Intermediate
Sixth-generation TPU with several times the per-chip compute of v5e and better energy efficiency. Teams moved inference from v5e to Trillium to serve more requests per chip.
Ironwood (TPU v7)
Google, announced April 2025 · Accelerator · Current · Intermediate
Inference-focused TPU with 192 GB HBM per chip, about 4.6 FP8 petaFLOPS and pods of up to 9,216 chips. A leading AI lab announced plans to use up to one million TPUs, including Ironwood, for its models.
TPU 8t and TPU 8i
Google, announced April 2026 · Accelerator · Emerging · Advanced
Eighth-generation TPUs split into a training chip (8t, 216 GB HBM3e) and an inference chip (8i, 288 GB HBM3e). TPU 8t superpods scale to 9,600 chips, while 8i adds an engine that speeds up collective operations when serving models.
Trainium2
AWS (Annapurna Labs), Trn2 instances from December 2024 · Accelerator · Current · Intermediate
AWS's second-generation training chip with 96 GB HBM; Trn2 UltraServers link 64 chips. AWS's Project Rainier cluster built with hundreds of thousands of Trainium2 chips serves a leading AI lab.
Trainium3
AWS, generally available December 2025 · Accelerator · Current · Intermediate
3 nm chip with 144 GB HBM3e and about 2.5 FP8 petaFLOPS; Trn3 UltraServers hold up to 144 chips. AWS claims a Trn3 UltraServer gives about 4.4 times the compute of a Trn2 UltraServer.
Inferentia2
AWS, Inf2 instances from 2023 · Accelerator · Current · Intermediate
AWS chip specialised for low-cost inference, with 32 GB HBM per chip. A chatbot company serves a 13-billion-parameter model on Inf2 instances to cut inference costs.
AWS Neuron SDK
AWS · Software · Current · Advanced
Compiler, runtime and tools that let PyTorch and JAX models run on Trainium and Inferentia. Developers trace a Hugging Face model with torch-neuronx before deploying to Inf2.
Maia 100
Microsoft, announced November 2023 · Accelerator · Legacy · Intermediate
Microsoft's first in-house AI accelerator, on TSMC 5 nm with 64 GB HBM2E, used internally in Azure. Microsoft used Maia 100 to learn how to build custom racks with sidekick liquid cooling.
Maia 200
Microsoft, January 2026 · Accelerator · Current · Intermediate
Inference accelerator on TSMC 3 nm with 216 GB HBM3e and native FP8/FP4, deployed in Azure. Microsoft said Maia 200 serves models including OpenAI GPT models inside Azure.
MTIA (Meta Training and Inference Accelerator)
Meta · Accelerator · Current · Intermediate
Meta's in-house chips, first for ranking and recommendation, with a published roadmap of further generations through 2027. MTIA chips help rank posts and ads in Facebook and Instagram feeds.
Custom AI ASIC design partners
Broadcom, Marvell, Alchip, GUC and others · Concept · Current · Advanced
Chip-design firms that help cloud companies turn their AI chip ideas into manufacturable silicon. Broadcom co-designs Google's TPUs and agreed in 2025 to build custom accelerators with OpenAI.

4.9 AI Chip Start-ups and Wafer-Scale Engines

Dozens of newer companies build AI chips with radically different designs from GPUs, hoping to be faster or cheaper for specific jobs such as answering chatbot questions. Some make one giant chip from a whole silicon wafer, some keep the entire model in ultra-fast on-chip memory, and some build chips that only run transformer models. The market moves fast: start-ups get funded, listed on stock markets, licensed or bought every year.

Start-up bets against the GPU

Big idea Memory strategy Sold as
Cerebras Whole-wafer chip 44 GB on-chip SRAM Systems and inference cloud
Groq Deterministic LPU Weights in on-chip SRAM GroqCloud API
SambaNova Reconfigurable dataflow SRAM + HBM + DDR tiers Racks and cloud
Tenstorrent Tensix + RISC-V cores GDDR6, open software Cards, servers, IP licences
Etched Transformer-only ASIC HBM Inference servers
d-Matrix Digital in-memory compute Compute inside SRAM PCIe inference cards

Most designs attack the memory-bandwidth limit of token-by-token generation.

Think of it like…

If GPUs are excellent all-purpose delivery vans, these start-ups build racing motorbikes, giant lorries or single-route trams: brilliant on their chosen route, less flexible elsewhere.

Real-world example

Priya builds a voice assistant for a fictional bank, Ledgerly, and needs answers to start within 300 milliseconds. Her GPU setup produces about 60 tokens per second per user. She tests an SRAM-heavy inference cloud that streams several hundred tokens per second per user, so replies feel instant. She keeps GPUs for fine-tuning, because the specialised service only runs a fixed menu of popular open models.

How it works

  1. Find a weakness of GPUs, such as slow off-chip memory during token-by-token generation.
  2. Design an architecture around it: wafer-scale chips, huge on-chip SRAM, dataflow layouts or transformer-only logic.
  3. Build a compiler that maps PyTorch or popular open models onto the new hardware.
  4. Sell systems or, increasingly, a cloud API that charges per token.
  5. Survive by winning a niche, being acquired or licensing the technology to a larger company.

Who uses it

AI infrastructure buyers, investors, hardware engineers and anyone comparing inference providers by speed and cost per token.

Key takeaway

Learn each start-up's one big idea: Cerebras = whole-wafer chip, Groq = deterministic SRAM LPU, SambaNova = reconfigurable dataflow, Tenstorrent = open RISC-V AI chips, Etched = transformer-only ASIC.

Going deeper

  • At batch size 1, generating each token reads every weight once, so speed ≈ memory bandwidth ÷ model bytes; SRAM-based designs exploit this with tens of TB/s of on-chip bandwidth.
  • The catch with SRAM-only designs is capacity: a large model must be split across many chips or wafers, so serving big models can need hundreds of chips.
  • Wafer-scale chips route around manufacturing defects with spare cores and need custom packaging, power delivery and water cooling for a single die of about 46,000 mm².
  • Business models are shifting from selling hardware to selling tokens: Cerebras, Groq and SambaNova all run inference APIs that compete on speed and price per million tokens.
  • Consolidation is common: SoftBank bought Graphcore, NVIDIA licensed Groq's technology and hired key staff, and others have pivoted or closed; software ecosystems remain the hardest barrier.
  • Edge AI start-ups such as Hailo, Axelera AI and DEEPX target cameras, robots and PCs at a few watts, where cost and power matter more than peak performance.

Common mistakes & misconceptions

  • Judging chips on single-user tokens per second only: throughput per dollar with many concurrent users may still favour GPUs.
  • Assuming any model runs on any start-up's cloud: most support a curated list of models and features.
  • Treating vendor benchmarks as neutral: check MLPerf results or independent tests with your own workload.

Try it: Why on-chip SRAM helps generation speed: a memory-bound tokens-per-second estimate

# Decode speed ~ memory bandwidth / bytes of weights read per token
params = 8e9            # 8B-parameter model
bytes_per_param = 1     # FP8
for name, bw in [('HBM3e GPU', 4.8e12), ('Hypothetical SRAM chip', 80e12)]:
    tps = bw / (params * bytes_per_param)
    print(f'{name}: ~{tps:,.0f} tokens/s upper bound (batch size 1)')

Key chips, architectures, standards, tools & concepts

Name What it does Example
Cerebras WSE-3
Cerebras Systems, 2024 · Accelerator · Current · Intermediate
Wafer-Scale Engine: one dinner-plate-sized chip with about 4 trillion transistors, 900,000 AI cores and 44 GB of on-chip SRAM. Cerebras's inference cloud serves open models at very high tokens per second from CS-3 systems.
Cerebras CS-3
Cerebras Systems · System · Current · Intermediate
Complete water-cooled system built around one WSE-3, clustered for training and inference. A national lab links multiple CS-3 systems to run large scientific simulations and AI models.
Cerebras IPO
Cerebras Systems, Nasdaq (CBRS), May 2026 · Concept · Current · Beginner
Cerebras listed on Nasdaq in May 2026, raising about $5.55 billion in the largest US tech IPO since 2019. Students following the industry can now read Cerebras's public financial filings to learn how AI chip businesses earn money.
Groq LPU
Groq · Accelerator · Current · Intermediate
Language Processing Unit: a deterministic chip that keeps model weights in on-chip SRAM to generate tokens extremely fast. GroqCloud serves open models through an API at hundreds of tokens per second per user.
NVIDIA-Groq licensing deal
NVIDIA and Groq, December 2025 · Concept · Current · Intermediate
Reported roughly $20 billion non-exclusive licence of Groq's inference technology, with Groq's founder and key staff joining NVIDIA. Groq continued as an independent company running GroqCloud, a structure analysts called a reverse acqui-hire.
SambaNova SN40L
SambaNova Systems · Accelerator · Current · Advanced
Reconfigurable Dataflow Unit with a three-tier memory of SRAM, HBM and DDR to hold many large models at once. An enterprise hosts dozens of fine-tuned expert models on one SambaNova rack and switches between them quickly.
Tenstorrent Wormhole and Blackhole
Tenstorrent (CEO Jim Keller) · Accelerator · Current · Intermediate
AI chips built from Tensix cores and RISC-V CPUs, using GDDR6 and built-in Ethernet, with open-source software. A developer buys a Blackhole PCIe card and programs it with the open TT-Metalium SDK.
Graphcore IPU
Graphcore (owned by SoftBank since 2024) · Accelerator · Current · Intermediate
Intelligence Processing Unit with thousands of independent cores and large on-chip memory, from a UK start-up. Graphcore's sale to SoftBank shows how hard it is for start-ups to compete with NVIDIA's software ecosystem.
Etched Sohu
Etched · Accelerator · Emerging · Intermediate
ASIC that hard-wires the transformer architecture, giving up flexibility for inference throughput. Etched raised large funding rounds in 2025-26 and says first Sohu systems are reaching early customers.
d-Matrix Corsair
d-Matrix · Accelerator · Emerging · Advanced
Inference PCIe card using digital in-memory computing, doing maths inside SRAM to cut data movement. A provider adds Corsair cards to existing servers to serve small models at low latency.
Hailo-8 and Hailo-10H
Hailo (Israel) · Accelerator · Current · Beginner
Low-power edge AI accelerators for cameras and embedded devices; Hailo-10H targets small generative models. The Raspberry Pi AI Kit uses a Hailo chip to run object detection on a hobby camera.
Wafer-scale integration
Concept; commercialised by Cerebras · Concept · Current · Intermediate
Using an entire silicon wafer as one chip instead of cutting it into many, routing around defective areas. A normal wafer yields dozens of GPU dies; Cerebras turns the same wafer into a single processor.
Dataflow architecture
General concept; used by SambaNova, Cerebras, Groq · Architecture · Current · Advanced
Design where data streams through a fixed layout of compute units, rather than instructions fetching data from memory. A compiler lays out each neural-network layer on a different area of the chip, like stations on an assembly line.

4.10 NPUs in PCs and Phones

An NPU (Neural Processing Unit) is a small, power-saving AI engine built into the main chip of modern laptops and phones. It runs features like background blur, live captions, photo enhancement and on-device assistants without draining the battery or sending data to the cloud. Its speed is usually quoted in TOPS, trillions of simple operations per second.

Microsoft's Copilot+ PC threshold is 40 TOPS; real app speed also depends on memory and software.
Laptop NPU ratings (vendor INT8 peak). Microsoft's Copilot+ PC threshold is 40 TOPS; real app speed also depends on memory and software.

Think of it like…

An NPU is like a calculator built into a cash register: it does one kind of sum very efficiently so the shop assistant (the CPU) can get on with everything else.

Real-world example

Jonah joins a video call on a Copilot+ laptop with a 45 TOPS NPU. Background blur, eye-contact correction and live captions run on the NPU using about 1-2 watts, so his CPU stays mostly idle and the fan stays quiet. When he ran the same effects on the CPU of his old laptop, the fans roared and the battery fell noticeably faster.

How it works

  1. An app asks the OS AI runtime (Windows ML, Core ML, Android NNAPI or LiteRT) to run a model.
  2. The runtime checks which engines exist and picks the NPU for supported operations.
  3. The model is usually quantised to INT8 or INT4 so it fits the NPU's strengths.
  4. The NPU's matrix arrays process each layer at low power; unsupported layers fall back to GPU or CPU.
  5. Results return to the app, and no data needs to leave the device.

Who uses it

Laptop and phone buyers, app developers adding on-device AI, and embedded engineers working with battery-powered devices.

Key takeaway

Know the Copilot+ PC bar — an NPU of at least 40 TOPS — and remember TOPS figures depend on precision, so compare like with like.

Going deeper

  • NPUs are built from multiply-accumulate arrays with local SRAM and DMA engines that run a pre-compiled graph of layers, very efficient at a few watts but less flexible than GPUs.
  • Models must be converted and quantised for each NPU with vendor tools (Qualcomm AI Hub and QNN, Intel OpenVINO, AMD Ryzen AI Software, Apple coremltools); unsupported operators fall back.
  • For on-device language models, memory bandwidth limits speed more than TOPS: a 3-billion-parameter 4-bit model must stream about 1.5 GB of weights for every generated token.
  • Phones spread AI across CPU, GPU, NPU, DSP and the image signal processor; camera pipelines run continuous AI on the ISP and NPU to save battery.
  • Windows ML, generally available since 2025, is built into Windows 11 and fetches vendor execution providers, so one app can target Qualcomm, Intel and AMD NPUs.

Common mistakes & misconceptions

  • Believing a higher TOPS number means faster apps: software support, precision and memory bandwidth often matter more.
  • Assuming every AI feature uses the NPU: many apps still run AI on the GPU or in the cloud.
  • Comparing INT8 TOPS with FP16 TFLOPS directly: they measure different precisions and are not equivalent.

Try it: See which AI execution providers ONNX Runtime can use on your PC

pip install onnxruntime        # or onnxruntime-qnn / onnxruntime-directml / onnxruntime-openvino
python -c "import onnxruntime as ort; print(ort.get_available_providers())"
# Windows: Task Manager > Performance shows an 'NPU' graph on Copilot+ PCs

Key chips, architectures, standards, tools & concepts

Name What it does Example
NPU (Neural Processing Unit)
Qualcomm, Intel, AMD, Apple, Google, MediaTek and others · Accelerator · Current · Beginner
Dedicated low-power block on a system-on-chip that runs neural-network layers, especially with INT8/INT4 maths. A phone's NPU removes background noise from calls in real time.
TOPS rating
Vendor marketing metric · Metric · Current · Beginner
Trillions of operations per second, usually at INT8; useful for rough comparison but not a direct measure of real app speed. Two laptops both rated 50 TOPS can differ greatly in real speed because of memory bandwidth and software.
Copilot+ PC
Microsoft, launched May 2024 · Standard · Current · Beginner
Windows PC category requiring an NPU of 40+ TOPS, at least 16 GB RAM and 256 GB storage, unlocking on-device AI features. Features like Recall, Live Captions translation and Cocreator need a Copilot+ PC's NPU.
Qualcomm Hexagon NPU
Qualcomm · Accelerator · Current · Intermediate
NPU in Snapdragon chips: about 45 TOPS in Snapdragon X Elite and up to 80 TOPS in Snapdragon X2 Elite. The first Copilot+ PCs in 2024 shipped with Snapdragon X Elite and its Hexagon NPU.
Intel NPU
Intel (Core Ultra processors) · Accelerator · Current · Intermediate
NPU built into Core Ultra chips; Lunar Lake's NPU 4 reaches about 48 TOPS, and Panther Lake's NPU 5 about 50. An office laptop with Core Ultra runs Teams background effects on the NPU.
AMD XDNA 2
AMD (from Xilinx AI Engine technology) · Architecture · Current · Intermediate
AMD's NPU architecture: about 50 TOPS in Ryzen AI 300 and up to about 60 TOPS in Ryzen AI 400. A Ryzen AI laptop qualifies as a Copilot+ PC thanks to its XDNA 2 NPU.
Apple Neural Engine
Apple (since A11 Bionic, 2017) · Accelerator · Current · Beginner
Apple's NPU in iPhone, iPad and Mac chips; the M4's 16-core Neural Engine is rated at 38 TOPS. Face ID and photo subject-lifting run on the Neural Engine.
Apple M5 GPU Neural Accelerators
Apple, October 2025 · Component · Current · Intermediate
M5 adds a matrix 'Neural Accelerator' inside each GPU core, boosting on-device AI beyond the Neural Engine. Running a local language model on an M5 MacBook uses these GPU accelerators for faster prompt processing.
Google Tensor G5 TPU
Google, Pixel 10 (2025) · Accelerator · Current · Intermediate
Phone chip made by TSMC on 3 nm with an upgraded on-device TPU that runs Gemini Nano. Pixel 10 features like on-device call summaries rely on the Tensor G5's TPU.
Core ML
Apple · Framework · Current · Intermediate
Apple's framework for running models on-device, automatically spreading work across CPU, GPU and Neural Engine. An iOS developer converts a PyTorch model with coremltools and ships it inside an app.
LiteRT (formerly TensorFlow Lite)
Google · Framework · Current · Intermediate
Lightweight runtime for on-device models on Android, embedded Linux and microcontrollers, with NPU and GPU delegates. A plant-identification app runs a 5 MB LiteRT model entirely on the phone.
Windows ML and ONNX Runtime
Microsoft · Framework · Current · Intermediate
Windows' built-in AI runtime, built on ONNX Runtime, that routes models to the best available CPU, GPU or NPU. A Windows app ships one ONNX model that runs on Qualcomm, Intel or AMD NPUs via vendor execution providers.
On-device small language models
Examples: Gemini Nano (Google), Phi Silica (Microsoft), Apple Foundation Models · Concept · Current · Beginner
Compact language models of a few billion parameters or fewer, quantised to run on NPUs privately and offline. Phone-based message summaries can run without an Internet connection using an on-device model.

4.11 FPGAs and Custom ASICs

An FPGA is a chip full of blank logic blocks and wires that you can rewire with a configuration file, so it can become almost any digital circuit. An ASIC is a chip designed for one job and fixed for ever at the factory, so it is the fastest and most efficient option but very expensive to create. Engineers describe both using hardware description languages such as Verilog and VHDL.

CPU/GPU vs FPGA vs ASIC

CPU / GPU FPGA ASIC
Flexibility Change software any time Rewire hardware in the field Fixed at manufacture
Up-front cost None beyond buying chips Low to moderate Tens of millions at leading nodes
Unit cost at volume Moderate High Lowest
Performance per watt Lowest Good Best
Time to deploy Days Weeks to months One to two years or more
Typical use General software, AI Telecom, trading, prototypes Phones, TPUs, crypto mining

The more fixed the hardware, the better its efficiency and unit cost, and the higher the up-front risk.

Think of it like…

An FPGA is a giant LEGO board you can rebuild into a new model whenever you like; an ASIC is a moulded plastic toy, perfect and cheap in volume, but you cannot change it.

Real-world example

Sanjay works at a fictional trading firm, Quickfill. Its software trading engine reacts in about 5 microseconds, so he implements the market-data parser on an FPGA network card and reaction time falls to under 1 microsecond. Later the firm considers turning the design into an ASIC but rejects it: a modern ASIC tape-out can cost tens of millions, and the strategy changes every few months, which only an FPGA can follow.

How it works

  1. Describe the circuit in Verilog, VHDL or SystemVerilog, or in C++ using high-level synthesis (HLS).
  2. Simulate it with test benches to check behaviour cycle by cycle.
  3. Synthesis turns the code into logic gates; place-and-route maps them onto the FPGA's blocks.
  4. The tool produces a bitstream that is loaded onto the FPGA, which can be reprogrammed later.
  5. For an ASIC, the verified design is turned into masks and manufactured (tape-out), with no changes afterwards.

Who uses it

FPGA and digital design engineers, chip verification engineers, telecom, defence, finance and embedded-systems companies.

Key takeaway

Remember the trade-off triangle: CPU = most flexible, FPGA = reconfigurable hardware, ASIC = fastest and most efficient but fixed and costly to design.

Going deeper

  • FPGA designs are limited by timing closure: every path between flip-flops must settle within one clock period, so typical FPGA designs run at a few hundred MHz, not multiple GHz.
  • Modern FPGAs add hard blocks such as DSP slices, block RAM, high-speed transceivers, PCIe and Ethernet controllers, Arm cores and even HBM, mixing fixed and programmable logic.
  • The ASIC flow runs RTL design, verification (often the biggest effort, with SystemVerilog and UVM), synthesis, physical design, sign-off and tape-out; leading-edge mask sets alone cost millions.
  • Prototyping ASICs on FPGAs is standard: large multi-FPGA emulation systems let software teams start work long before first silicon returns.
  • Microsoft's Project Catapult deployed FPGAs across its data-centre servers to accelerate Bing search and Azure networking, a well-known large-scale FPGA deployment.
  • Open-source silicon is growing: the SkyWater 130 nm open PDK, OpenROAD and community shuttle programmes let students tape out small chips cheaply.

Common mistakes & misconceptions

  • Writing HDL like software: Verilog describes parallel hardware, so every statement becomes circuitry operating at the same time.
  • Skipping simulation and testing on the board: bugs are far harder to find in hardware than in a test bench.
  • Choosing an ASIC too early: if requirements change, a fixed chip becomes an expensive mistake.

Try it: A tiny Verilog module: an 8-bit counter (simulate with Icarus Verilog or any online simulator)

module counter (
  input  wire       clk,
  input  wire       reset,
  output reg  [7:0] count
);
  always @(posedge clk) begin
    if (reset) count <= 8'd0;
    else       count <= count + 8'd1;   // wraps after 255
  end
endmodule

Key chips, architectures, standards, tools & concepts

Name What it does Example
FPGA (Field-Programmable Gate Array)
AMD (Xilinx), Altera, Lattice, Microchip · Accelerator · Current · Beginner
Chip of configurable logic blocks, memory and DSP slices joined by programmable wiring, reprogrammable after manufacture. A 5G radio unit uses an FPGA so its signal processing can be updated as standards evolve.
LUTs, flip-flops and DSP slices
FPGA building blocks · Component · Current · Intermediate
Look-up tables implement small logic functions, flip-flops store bits, and DSP slices do fast multiply-add maths. A filter design reports using 12,000 LUTs and 64 DSP slices of the target FPGA.
ASIC (Application-Specific Integrated Circuit)
Designed by chip companies; built by foundries such as TSMC, Samsung, Intel Foundry · Concept · Current · Beginner
A chip designed for a single purpose, giving top speed and efficiency but with high one-off design costs. Bitcoin mining chips, Google's TPUs and phone modems are all ASICs.
AMD Versal Adaptive SoC
AMD (formerly Xilinx; acquisition completed 2022) · Accelerator · Current · Intermediate
Device combining FPGA fabric, Arm CPUs and AI Engine vector processors on one chip. A car maker uses Versal AI Edge chips for camera and radar processing in driver-assistance systems.
AMD Alveo cards
AMD · Accelerator · Current · Intermediate
Data-centre accelerator cards based on FPGAs or adaptive SoCs for networking, finance and video. A trading firm deploys Alveo cards to process market data within nanoseconds.
Altera Agilex
Altera (51% owned by Silver Lake, 49% by Intel, since September 2025) · Accelerator · Current · Intermediate
Altera's current FPGA families (Agilex 3, 5, 7 and 9), from low-cost to high-end. A broadcaster uses Agilex FPGAs to switch and convert high-resolution video streams in real time.
Lattice FPGAs
Lattice Semiconductor · Accelerator · Current · Intermediate
Small, low-power FPGAs (Nexus, Avant families) for edge AI, sensors and system control. A laptop uses a tiny Lattice FPGA to manage sensors and power sequencing.
Microchip PolarFire
Microchip Technology · Accelerator · Current · Intermediate
Flash-based, low-power FPGAs, including radiation-tolerant versions used in space. A satellite payload uses a radiation-tolerant PolarFire FPGA for image processing.
Verilog and SystemVerilog
IEEE 1364 (Verilog); IEEE 1800 (SystemVerilog) · Language · Current · Intermediate
Hardware description languages for designing and verifying digital circuits; SystemVerilog adds powerful verification features. Chip verification engineers write SystemVerilog test benches with UVM to test designs.
VHDL
IEEE 1076 · Language · Current · Intermediate
Strongly typed hardware description language, popular in Europe, aerospace and defence. A railway signalling FPGA is written in VHDL to meet strict safety-review practices.
High-level synthesis (HLS)
AMD Vitis HLS; Siemens Catapult; others · Technology · Current · Advanced
Tools that convert C or C++ functions into hardware descriptions, so software developers can design FPGA logic. An engineer writes an image filter in C++ and Vitis HLS produces a pipelined hardware block.
Vivado, Vitis and Quartus Prime
AMD (Vivado, Vitis); Altera (Quartus Prime) · Tool · Current · Beginner
Vendor design suites for synthesis, place-and-route, timing analysis and bitstream generation. Students build their first blinking-LED design in Vivado on a low-cost FPGA board.
Open-source FPGA toolchain (Yosys, nextpnr)
YosysHQ and open-source community · Tool · Current · Intermediate
Free tools for synthesis and place-and-route that support several FPGA families, great for learning. A hobbyist programs a Lattice iCE40 board entirely with open-source tools.
Tape-out
Semiconductor industry term · Concept · Current · Intermediate
The point where a finished chip design is sent for mask making and manufacture, after which changes are very costly. After tape-out, a team waits weeks to months for first silicon to return from the foundry.

4.12 How a Chip Is Designed: From RTL to Tape-Out

Before a chip is manufactured, engineers describe its logic in a hardware description language, prove it works with simulations and other checks, and turn it into a physical layout of billions of transistors and wires. Specialised software called EDA (electronic design automation) does much of the heavy lifting. The finished layout goes to a foundry at 'tape-out', and the first chips that come back are tested in the lab.

The later a bug is found, the more it costs — after tape-out it can mean new masks and months of delay.
The digital chip design flow. The later a bug is found, the more it costs — after tape-out it can mean new masks and months of delay.

Think of it like…

Designing a chip is like building a new city: architects write the zoning plan (RTL), inspectors check every rule (verification), planners place buildings and roads (place and route), and only when every permit passes do the bulldozers start (tape-out).

Real-world example

Kestrel Silicon, a fictional start-up, is building a small AI inference chip on a 5 nm process. Its RTL engineers write SystemVerilog for a matrix engine, while a larger verification team tests it with UVM simulations and formal tools, and an FPGA prototype runs real models months before silicon exists. Physical designers synthesise, place and route the design until timing closes at 1.5 GHz. After tape-out, the first wafers return a few months later for bring-up in the lab.

How it works

  1. Architects write a specification and a performance model, often in C++ or Python, to choose features, sizes and power targets.
  2. RTL designers describe the logic in SystemVerilog or VHDL as registers and the logic between them.
  3. Verification engineers simulate the RTL with randomised tests, coverage tracking and formal proofs, and run real software on emulators or FPGA prototypes.
  4. Synthesis maps the RTL to a netlist of standard cells from the foundry's library; place and route positions those cells, builds the clock tree and wires everything up.
  5. Sign-off checks timing, power, signal integrity and manufacturing rules (DRC, LVS) before the layout files go to the foundry at tape-out.
  6. Returned chips go through bring-up, post-silicon validation and production testing; serious bugs may need a respin.

Who uses it

RTL design, design verification, physical design, DFT and post-silicon validation engineers, EDA tool developers and chip architects at chip companies, hyperscalers and start-ups follow this flow.

Key takeaway

Know the proportions: verification usually takes more engineering effort than writing the design, and a mask set at a leading-edge node can cost tens of millions of dollars, so first-time-right silicon matters.

Going deeper

  • Three companies dominate commercial EDA — Synopsys (which completed its acquisition of simulation firm Ansys in 2025), Cadence and Siemens EDA — covering simulation, synthesis, place and route and sign-off, plus licensable IP blocks.
  • A PDK (process design kit) from the foundry contains design rules, transistor models and libraries of standard cells, memories and I/O; designers cannot target a process without one.
  • Timing closure means every path between flip-flops meets setup and hold times at every process, voltage and temperature corner; static timing analysis checks billions of paths without simulation.
  • Verification combines SystemVerilog UVM testbenches, coverage metrics that expose untested behaviour, formal property checking, and hardware emulators such as Palladium, ZeBu and Veloce that boot operating systems before silicon exists.
  • Design for test inserts scan chains and built-in self-test so automatic test equipment can screen every die for manufacturing defects in seconds.
  • Open-source flows such as OpenROAD and OpenLane, open PDKs such as SkyWater SKY130, and shared-wafer programmes such as Tiny Tapeout let students take real designs to silicon cheaply.
  • Large chips are now assembled from chiplets, adding package design, die-to-die links such as UCIe, and multi-die timing, power and thermal analysis to the flow.

Common mistakes & misconceptions

  • Treating RTL like software: it describes parallel hardware, so every always block runs at the same time and a loop becomes physical copies of logic.
  • Leaving verification until the end: it runs alongside design from day one, because a bug found after tape-out can force a costly respin.
  • Assuming a design that simulates correctly will run at the target clock: only static timing analysis on the placed-and-routed layout proves timing.

Try it: Simulate a tiny counter with open-source Icarus Verilog

// counter.v - a 4-bit counter with synchronous reset
module counter(input clk, input rst, output reg [3:0] q);
  always @(posedge clk) q <= rst ? 4'd0 : q + 4'd1;
endmodule

# shell: compile with a testbench you write, then run the simulation
# iverilog -o sim counter.v counter_tb.v && vvp sim

Key chips, architectures, standards, tools & concepts

Name What it does Example
RTL (register-transfer level)
IEEE 1800 SystemVerilog; IEEE 1076 VHDL · Concept · Current · Intermediate
A level of hardware description showing registers and the logic that moves and transforms data between them on each clock edge. 'always_ff @(posedge clk) count <= count + 1;' is RTL describing a counter register and the adder feeding it.
Functional verification and UVM
Accellera UVM; IEEE 1800.2 · Concept · Current · Intermediate
Checking the RTL behaves as specified through simulation; UVM is the standard SystemVerilog method for reusable, randomised testbenches. A cache-controller testbench fires millions of random reads and writes and checks every result against a reference model.
Formal verification
EDA tools (Cadence Jasper, Synopsys VC Formal, Siemens Questa Formal) · Concept · Current · Advanced
Mathematically proving that properties hold for every possible input sequence, rather than testing chosen examples. A formal tool proves a bus arbiter can never grant two requesters at once, a corner case random tests might miss.
Hardware emulation and FPGA prototyping
Cadence Palladium; Synopsys ZeBu; Siemens Veloce; FPGA boards · Technology · Current · Advanced
Running the RTL on special hardware at megahertz speeds so firmware, drivers and operating systems can run before silicon exists. A phone-chip team boots Android on an emulator months before tape-out, finding driver and hardware bugs together.
Logic synthesis
EDA tools (Synopsys Fusion Compiler, Cadence Genus, open-source Yosys) · Concept · Current · Intermediate
Automatically converting RTL into a netlist of gates and flip-flops from a standard-cell library, optimised for speed, power and area. Synthesis turns a 32-bit adder description into hundreds of standard cells sized to meet a 2 GHz clock.
Standard cell library
Foundries and IP vendors · Component · Current · Intermediate
A catalogue of pre-designed, characterised gates and flip-flops of fixed height, placed in rows to build digital logic. A library offers the same NAND gate in several drive strengths and threshold voltages, trading speed against leakage power.
PDK (process design kit)
Foundries (TSMC, Samsung Foundry, Intel Foundry, SkyWater and others) · Resource · Current · Intermediate
The foundry's package of design rules, device models and libraries needed to design for one specific manufacturing process. Students design with the open SkyWater SKY130 PDK, while commercial teams sign agreements to access leading-edge PDKs.
Place and route (PnR)
EDA tools (Synopsys Fusion Compiler, Cadence Innovus, OpenROAD) · Concept · Current · Intermediate
Floorplanning blocks, placing millions of cells, building the clock tree and routing wires across many metal layers. Engineers move memory blocks closer to the matrix engine to shorten wires and fix failing timing paths.
Static timing analysis (STA)
EDA tools (Synopsys PrimeTime, Cadence Tempus, OpenSTA) · Concept · Current · Advanced
Checking that every logic path meets setup and hold times at all process, voltage and temperature corners, without simulation. STA reports a path missing 1.5 GHz by 20 picoseconds, so the team adds a pipeline register to fix it.
Physical verification (DRC and LVS)
EDA tools (Siemens Calibre, Synopsys IC Validator, Cadence Pegasus) · Concept · Current · Advanced
DRC checks the layout against manufacturing rules; LVS confirms the layout matches the intended circuit netlist. Tape-out waits until the final DRC run reports zero violations across billions of layout shapes.
Design for test (DFT) and scan chains
EDA tools; IEEE 1149.1 (JTAG); IEEE 1500 · Concept · Current · Advanced
Adding structures that link flip-flops into shift chains, plus self-test logic, so every manufactured chip can be checked quickly. A factory tester shifts patterns through scan chains in seconds and rejects dies with manufacturing defects.
EDA vendors (Synopsys, Cadence, Siemens EDA)
Synopsys; Cadence Design Systems; Siemens · Organisation · Current · Beginner
The three companies supplying most commercial chip-design software, plus licensable IP blocks such as PCIe and DDR controllers. A start-up licenses a PCIe controller and PHY from an EDA vendor rather than designing its own.
Open-source chip flows (OpenROAD, OpenLane, Tiny Tapeout)
Open-source community; Tiny Tapeout project · Tool · Current · Beginner
Free toolchains and shared-wafer programmes that let students and hobbyists turn Verilog into real manufactured chips. A student submits a small design to Tiny Tapeout and, months later, receives a working chip for a modest fee.
Post-silicon validation and bring-up
Chip companies · Concept · Current · Advanced
Testing the first chips in the lab: powering up, booting firmware, running stress tests and hunting bugs that simulation missed. Bring-up engineers find a rare cache bug and work around it with a firmware patch instead of a respin.
Respin and steppings
Chip industry practice · Concept · Current · Intermediate
Manufacturing a corrected chip revision after bugs or yield problems, costing new masks and months; revisions are labelled as steppings. A chip's first silicon is stepping A0; after fixes, a B0 stepping often becomes the production version.

4.13 Number Formats for AI: FP64 to FP4 and Quantisation

Computers store numbers in fixed-size formats; fewer bits means less memory and faster maths, but less precision. Scientific simulation needs 64-bit numbers, but AI models work well with 16, 8 or even 4 bits per number. Each new AI chip generation adds smaller formats, and quantisation is the art of shrinking a model's numbers without making it noticeably worse.

Each halving of bits halves the memory; block-scaled formats add a few percent for their scales.
Weight memory for a 70B model by format. Each halving of bits halves the memory; block-scaled formats add a few percent for their scales.

Think of it like…

Choosing a number format is like choosing how precisely to measure ingredients: a pharmacist needs milligram scales (FP64), but a pizza chef is fine with 'a handful' (FP4) and works far faster.

Real-world example

Elena has a 70-billion-parameter model. In BF16 (2 bytes per number) the weights take about 140 GB, needing two 80 GB GPUs. Quantised to FP8 they need about 70 GB and fit on one H200, and in 4-bit about 35 GB, small enough for a single 48 GB workstation card. Her evaluation shows the 4-bit version loses about one point of accuracy on her test set, which she accepts for her customer-support bot.

How it works

  1. A floating-point number has a sign bit, exponent bits (range) and mantissa bits (precision).
  2. Training usually uses BF16 or FP8 maths while keeping a master copy of weights in FP32 (mixed precision).
  3. Block-scaled formats (MXFP8, MXFP4, NVFP4) share one scale factor across small groups of values to keep accuracy.
  4. After training, quantisation converts weights, and sometimes activations, to INT8, FP8, INT4 or FP4.
  5. Always re-test model quality after changing precision.

Who uses it

ML engineers, inference-optimisation specialists, HPC scientists and chip architects choose formats daily.

Key takeaway

Memorise the bit layouts: FP32 = 1/8/23, FP16 = 1/5/10, BF16 = 1/8/7, FP8 E4M3 and E5M2, FP4 E2M1 — and remember BF16 keeps FP32's range.

Going deeper

  • Bit layouts (sign/exponent/mantissa): FP32 1/8/23, FP16 1/5/10, BF16 1/8/7, TF32 1/8/10, FP8 E4M3 and E5M2, FP6 E2M3 and E3M2, FP4 E2M1. Exponent bits set range; mantissa bits set precision.
  • Each halving of bit width roughly doubles tensor-core throughput and halves memory traffic, which is why inference moved from FP16 to FP8 and now FP4 on Blackwell-class chips.
  • Block scaling shares one scale per small block: MX formats use an 8-bit power-of-two scale per 32 values, while NVFP4 uses an FP8 E4M3 scale per 16 values plus a per-tensor FP32 scale.
  • Activations and the KV cache can be quantised too, for example an FP8 KV cache, which often saves more memory than weight quantisation in long-context serving.
  • Outliers are the enemy of low precision: a few huge activation values stretch the scale, so methods such as SmoothQuant and AWQ rebalance them before quantising.
  • Scientific HPC still needs FP64; some AI-focused GPUs cut FP64 throughput, so supercomputing centres check FP64 figures or use FP64 emulation on tensor cores.

Common mistakes & misconceptions

  • Training in FP16 without loss scaling: small gradients underflow to zero and the model silently stops learning.
  • Assuming 4-bit is always good enough: reasoning, maths and coding tasks can degrade noticeably, so evaluate on your own tasks.
  • Comparing TFLOPS at different precisions: an FP4 figure is not comparable with a BF16 or FP64 figure.
  • Confusing FP16 with BF16: both use 16 bits, but BF16 keeps FP32's range while FP16 overflows above 65,504.

Try it: Compare the range and precision of formats in PyTorch

import torch
for dt in [torch.float64, torch.float32, torch.bfloat16, torch.float16,
           torch.float8_e4m3fn, torch.float8_e5m2]:
    f = torch.finfo(dt)
    print(f'{str(dt):22} bits={f.bits:2}  max={f.max:.3g}  eps={f.eps:.3g}')

x = torch.tensor(0.1)
print(x.to(torch.bfloat16).item(), x.to(torch.float16).item())

Key chips, architectures, standards, tools & concepts

Name What it does Example
TF32
NVIDIA, introduced with Ampere (2020) · Format · Current · Intermediate
Tensor-core mode using FP32's 8-bit exponent with a 10-bit mantissa, speeding up FP32 code with no changes. PyTorch can use TF32 for matrix multiplies on A100 and newer GPUs when allowed by a setting.
BF16 (bfloat16)
Google Brain; supported by most AI chips · Format · Current · Beginner
The default AI training format: FP32's 8-bit exponent with a 7-bit mantissa, so it rarely overflows and needs no loss scaling. Most large language models today are trained with BF16 maths.
FP8 E4M3 and E5M2
NVIDIA, Arm and Intel proposal; OCP 8-bit floating point spec · Format · Current · Intermediate
AI-focused 8-bit floats: E4M3 (more precision, max 448) for weights and activations, E5M2 (more range, max 57,344) for gradients. Hopper's Transformer Engine picks FP8 formats per layer and rescales to keep training stable.
MX formats (MXFP8, MXFP6, MXFP4)
Open Compute Project Microscaling Formats spec v1.0 (2023) · Standard · Current · Advanced
Block-scaled formats where 32 values share one power-of-two scale factor, keeping small numbers accurate at low bit widths. AMD's MI355X and NVIDIA's Blackwell support MXFP4 and MXFP6 in hardware.
NVFP4
NVIDIA, Blackwell (2024-25) · Format · Current · Advanced
4-bit float format using blocks of 16 values with an FP8 scale plus a per-tensor scale, for accurate 4-bit inference. NVIDIA quotes Rubin's headline inference figure in NVFP4 petaFLOPS.
FP4 (E2M1)
OCP MX spec; vendor implementations · Format · Current · Advanced
4-bit float that can only represent values such as 0, 0.5, 1, 1.5, 2, 3, 4 and 6 (plus sign), so scaling is vital. Without a block scale, a weight of 0.02 would round to zero in plain FP4.
INT8
Integer format, widely supported · Format · Current · Beginner
8-bit integers with a scale factor, long used for efficient inference on CPUs, GPUs and NPUs. A phone camera model is quantised to INT8 to run on the NPU.
INT4
Integer format; GPTQ, AWQ and other quantisation methods · Format · Current · Intermediate
4-bit integers for weights, cutting model memory to a quarter of FP16 for local inference. A laptop runs a 4-bit 8-billion-parameter model in about 5 GB of memory.
Quantisation (PTQ and QAT)
General ML technique · Technology · Current · Intermediate
Converting a model to lower precision after training (post-training quantisation) or while training (quantisation-aware training). A team uses PTQ to make an FP8 copy of its model in an hour, then checks accuracy on a benchmark.
Mixed-precision training
Micikevicius et al. (NVIDIA/Baidu), 2017 · Technology · Current · Intermediate
Doing most maths in low precision while keeping sensitive values, such as weight updates, in FP32. PyTorch's torch.autocast runs layers in BF16 while the optimiser updates FP32 weights.
2:4 structured sparsity
NVIDIA, Ampere and later · Technology · Current · Advanced
Pattern where two of every four weights are zero, letting tensor cores skip them for up to twice the throughput. Vendors often quote 'with sparsity' figures that are double the dense numbers, so read datasheets carefully.
FP6 (E2M3 and E3M2)
OCP Microscaling (MX) specification · Format · Current · Advanced
6-bit floats between FP8 and FP4 in size and accuracy, supported in hardware by Blackwell and MI350-class accelerators. A team chooses MXFP6 when FP4 loses too much accuracy on its model but FP8 uses too much memory.
Loss scaling
Mixed-precision training practice (PyTorch GradScaler) · Technology · Current · Intermediate
Multiplying the loss by a large factor before backpropagation so tiny FP16 gradients do not underflow to zero, then dividing it back. PyTorch's GradScaler raises or lowers the scale automatically and skips any step that produces infinities.
GPTQ and AWQ
GPTQ (Frantar et al., 2022); AWQ (Lin et al., MIT, 2023) · Technology · Current · Advanced
Popular post-training methods that quantise language-model weights to about 4 bits, using calibration data to preserve accuracy. A team downloads an AWQ 4-bit version of an open model and serves it with vLLM on a single GPU.
GGUF
ggml / llama.cpp project · Format · Current · Intermediate
File format for quantised models used by llama.cpp and tools such as Ollama, offering 2- to 8-bit variants like Q4_K_M. A student downloads a Q4_K_M GGUF of an 8-billion-parameter model, about 5 GB, and chats with it offline on a laptop.

4.14 AI Training vs Inference: Compute and Memory Maths

Training teaches a model by showing it huge amounts of text or images and adjusting its numbers; inference is using the finished model to answer questions. Training is a giant one-off job measured in total FLOPs and GPU-hours, while inference runs all day for every user and is measured in tokens per second, response time and cost per token. Simple rules of thumb let you estimate both before spending any money.

Training vs inference at a glance

Training Inference
Goal Learn weights from data Answer users with fixed weights
Compute ≈ 6 × N × D FLOPs in total ≈ 2 × N FLOPs per token
Memory per parameter ~16 bytes (Adam, mixed precision) 1-2 bytes plus KV cache
Main bottleneck Compute and interconnect Memory bandwidth (decode)
Key metrics GPU-hours, MFU TTFT, tokens/s, cost per token
Duration Weeks, one-off Continuous, every request

Training is a huge one-off compute job; inference is a never-ending memory-bandwidth job.

Think of it like…

Training is writing and printing an encyclopedia once (enormous effort); inference is people looking things up in it millions of times a day (each lookup is cheap, but they never stop).

Real-world example

Ravi plans to train a 7-billion-parameter model on 1 trillion tokens. Using 6 × N × D, that needs about 4.2 × 10²² FLOPs. An H100 offers about 989 teraFLOPS of dense BF16; at a realistic 40% utilisation that is about 4 × 10¹⁴ useful FLOPs per second, so the job needs roughly 29,500 GPU-hours — about 15 days on 80 GPUs. For serving, he budgets 14 GB for BF16 weights plus memory for each user's KV cache.

How it works

  1. Training compute ≈ 6 × parameters × training tokens (forward plus backward pass).
  2. Inference compute ≈ 2 × parameters FLOPs per generated token.
  3. Memory for weights = parameters × bytes per parameter; training with Adam needs roughly 16 bytes per parameter plus activations.
  4. Inference adds a KV cache per user that grows with context length.
  5. Prefill (reading the prompt) is compute-bound; decode (writing tokens one by one) is memory-bandwidth-bound, so batching users together raises throughput.

Who uses it

ML engineers, AI infrastructure planners, cloud cost analysts and researchers sizing GPU clusters use these estimates daily.

Key takeaway

Memorise 6ND for training, 2N per token for inference, and '2 bytes per parameter in BF16' — together they answer most interview sizing questions.

Going deeper

  • Worked sizing: a 70B model in BF16 needs 140 GB of weights; with grouped-query attention its KV cache is about 2.5 GB per user at 8k context, so 32 users add roughly 80 GB.
  • Decode ceiling at batch size 1 ≈ memory bandwidth ÷ weight bytes: an H200 (about 4.8 TB/s) serving a 70 GB FP8 model tops out near 68 tokens per second per sequence.
  • Reasoning models shift spending toward inference: 'test-time compute' generates many hidden tokens per answer, so serving now dominates many AI compute budgets.
  • PagedAttention in vLLM stores the KV cache in fixed-size blocks, like virtual-memory pages, cutting fragmentation so many more concurrent requests fit on each GPU.
  • Large training runs are limited by failures as well as FLOPs: with tens of thousands of GPUs, hardware faults happen frequently, so jobs checkpoint often and restart automatically.
  • MFU counts only the model's useful FLOPs, while hardware FLOPs utilisation (HFU) also counts recomputation, so check which one a paper or vendor reports.

Common mistakes & misconceptions

  • Sizing inference by weights alone: the KV cache for many users and long contexts can exceed the model's own memory.
  • Using peak FLOPS in training estimates: real MFU of 30-50% can double or triple the time and cost.
  • Maximising tokens per second per GPU while ignoring latency: users notice slow time-to-first-token even when throughput looks great.

Try it: Estimate training GPU-hours and KV-cache size in Python

N, D = 7e9, 1e12                  # parameters, training tokens
flops = 6 * N * D
gpu_flops = 989e12 * 0.40          # H100 dense BF16 at 40% MFU
print(f'Training: {flops:.2e} FLOPs = {flops/gpu_flops/3600:,.0f} GPU-hours')

# KV cache per token = 2 (K and V) x layers x kv_heads x head_dim x bytes
layers, kv_heads, head_dim, bytes_ = 32, 8, 128, 2
per_tok = 2 * layers * kv_heads * head_dim * bytes_
print(f'KV cache: {per_tok/1024:.0f} KiB/token, {per_tok*8192/2**30:.1f} GiB for 8k context')

Key chips, architectures, standards, tools & concepts

Name What it does Example
Training vs inference
General ML concept · Concept · Current · Beginner
Training adjusts a model's weights using data and gradients; inference runs the fixed model to produce outputs for users. A lab trains once for weeks on 10,000 GPUs, then serves the model to millions of users for months.
Parameters (N)
General ML concept · Concept · Current · Beginner
The learned numbers (weights) inside a model; model size is usually quoted in billions of parameters. An '8B' model has about 8 billion parameters, roughly 16 GB in BF16.
Tokens (D)
General ML concept · Concept · Current · Beginner
Chunks of text, roughly three-quarters of an English word each, that models read and generate. A 1,000-word essay is roughly 1,300 tokens.
6ND training-compute rule
Kaplan et al. (OpenAI), Scaling Laws, 2020 · Concept · Current · Intermediate
Training a dense transformer costs about 6 FLOPs per parameter per training token: 2 forward and 4 backward. A 70B model trained on 15 trillion tokens needs about 6.3 × 10²⁴ FLOPs.
Chinchilla scaling
Hoffmann et al. (DeepMind), 2022 · Concept · Current · Intermediate
Finding that compute-optimal training uses roughly 20 tokens per parameter; many modern models now train on far more. A compute-optimal 10B model would see about 200 billion tokens.
Training memory (weights, gradients, optimiser)
Rule of thumb from ZeRO paper (Rajbhandari et al., 2020) · Concept · Current · Advanced
Mixed-precision Adam training needs about 16 bytes per parameter before activations, far more than inference. Training a 7B model needs about 112 GB just for weights and optimiser state, so it must be split across GPUs.
KV cache
Transformer inference technique · Concept · Current · Intermediate
Stored attention keys and values for every past token, so the model does not recompute them; grows with users and context length. Serving 100 users with 8,000-token contexts can need more memory for KV cache than for the weights.
Prefill and decode
LLM inference phases · Concept · Current · Intermediate
Prefill processes the whole prompt in parallel (compute-bound); decode generates one token at a time (memory-bound). Pasting a 50-page document makes prefill slow, while a short question spends most time in decode.
Batching and continuous batching
Popularised by vLLM (UC Berkeley, 2023) and others · Technology · Current · Intermediate
Processing many users' requests together so each weight read from memory serves many tokens, adding new requests as others finish. Batching 32 users raises total tokens per second many times with only a small delay per user.
TTFT and tokens per second
LLM serving metrics · Metric · Current · Beginner
Time to first token measures responsiveness; tokens per second per user and per GPU measure speed and efficiency. A chat service targets TTFT under 500 ms and at least 50 tokens per second per user.
MFU (Model FLOPs Utilisation)
Introduced in Google's PaLM paper, 2022 · Metric · Current · Advanced
The share of a chip's peak FLOPS that does useful model maths; 30-50% is typical for large training jobs. A team improves MFU from 32% to 45% by overlapping communication with computation.
vLLM, SGLang and TensorRT-LLM
vLLM and SGLang (open-source communities); TensorRT-LLM (NVIDIA) · Framework · Current · Intermediate
High-performance LLM serving engines with paged KV cache, continuous batching and quantised kernels. A start-up serves an open model with vLLM's OpenAI-compatible API on four GPUs.
FlashAttention
Tri Dao et al., 2022 onward · Technology · Current · Advanced
Attention algorithm that tiles work in on-chip memory to avoid writing huge intermediate matrices to HBM. Switching to FlashAttention lets a model handle much longer contexts on the same GPU.
Disaggregated serving
NVIDIA Dynamo, llm-d and others · Technology · Emerging · Advanced
Running prefill and decode on separate pools of GPUs so each phase gets hardware suited to it. A provider uses compute-heavy GPUs for prefill and memory-rich GPUs for decode, cutting cost per token.
MoE: active vs total parameters
Mixture-of-experts models such as Mixtral and DeepSeek-V3 · Concept · Current · Intermediate
Mixture-of-experts models route each token through a few experts, so compute follows active parameters while memory must hold all of them. DeepSeek-V3 has 671 billion parameters but activates about 37 billion per token, so its compute per token resembles a 37B dense model.
Speculative decoding
Leviathan et al. (Google) and Chen et al. (DeepMind), 2023 · Technology · Current · Advanced
A small draft model proposes several tokens that the large model verifies in one pass, speeding generation without changing outputs. A service drafts with a 1B model and verifies with its 70B model, generating tokens two to three times faster.

4.15 Connecting Accelerators: Scale-Up and Scale-Out

Big AI models need hundreds or thousands of accelerators working as one, so the links between chips matter as much as the chips themselves. Scale-up links join a few dozen GPUs so closely that they can share memory like one giant GPU; scale-out networks join many such groups across a data centre. Standards such as NVLink, UALink, InfiniBand and Ultra Ethernet compete to provide these links.

NVLink joins the GPUs inside each rack; every GPU's NIC also joins a rail of the InfiniBand or Ethernet fabric.
Scale-up inside racks, scale-out between them. NVLink joins the GPUs inside each rack; every GPU's NIC also joins a rail of the InfiniBand or Ethernet fabric.

Think of it like…

Scale-up is like colleagues sitting around one table passing papers instantly; scale-out is like many offices connected by a fast courier service — still quick, but each trip takes longer.

Real-world example

Hannah's fictional lab, Orbitar AI, trains a large model across 1,152 GPUs in 16 NVL72 racks. Inside each rack, NVLink lets 72 GPUs exchange tensor-parallel data at about 1.8 TB/s per GPU. Between racks, each GPU has an 800 Gb/s network port on an InfiniBand fabric for data-parallel gradient averaging. When one optical cable fails, NCCL logs a slowdown and the job manager restarts from the latest checkpoint.

How it works

  1. Inside a server or rack, GPUs connect through scale-up links (NVLink, Infinity Fabric, UALink) and switches.
  2. Each GPU also gets a high-speed network card (often 400-800 Gb/s) for scale-out.
  3. Scale-out fabrics (InfiniBand or RDMA Ethernet) link thousands of GPUs, usually in a rail-optimised fat-tree topology.
  4. GPUDirect RDMA lets network cards read and write GPU memory directly, skipping the CPU.
  5. Libraries like NCCL run collectives such as all-reduce, matching parallelism (tensor, pipeline, data) to the fastest links.

Who uses it

AI infrastructure engineers, network engineers in AI data centres, HPC architects and distributed-training specialists.

Key takeaway

Remember: tensor parallelism stays inside the scale-up domain (fastest links), while data parallelism can cross the scale-out network.

Going deeper

  • The bandwidth gap shapes parallelism: NVLink 5 gives each GPU about 900 GB/s per direction, while an 800 Gb/s NIC gives 100 GB/s, roughly nine times less.
  • Ring all-reduce sends about 2 × (n−1)/n times the data size per GPU, so gradient sync time depends on message size ÷ link bandwidth plus latency terms for small messages.
  • AI fabrics are usually non-blocking fat trees with rail optimisation; large clusters need three switch tiers, and topology-aware schedulers keep jobs within as few hops as possible.
  • Ethernet for AI needs congestion control: RoCE v2 relies on PFC and ECN-based schemes such as DCQCN, while Ultra Ethernet adds packet spraying and new transport to avoid hot spots.
  • Scale-up domains keep growing: 8 GPUs per HGX server, 72 in NVL72 racks and several hundred GPU dies in planned Rubin Ultra racks, while UALink targets up to 1,024 accelerators.
  • Co-packaged optics place lasers beside the switch chip to save power compared with pluggable transceivers; NVIDIA and Broadcom have announced CPO switches for AI networks.

Common mistakes & misconceptions

  • Running tensor parallelism across racks: it needs the fastest links, so keep it inside one NVLink or scale-up domain.
  • Treating RoCE Ethernet and InfiniBand as drop-in equals: RoCE needs careful tuning of PFC, ECN and buffers to avoid stalls.
  • Ignoring one slow link or GPU: collective operations run at the speed of the slowest participant in the job.

Try it: Show how GPUs and network cards are connected in your server

nvidia-smi topo -m        # matrix: NV# = NVLink, PIX/PXB = PCIe, SYS = across CPUs
nvidia-smi nvlink -s      # NVLink link status and speed
ibstat                    # InfiniBand ports (if present)

Key chips, architectures, standards, tools & concepts

Name What it does Example
Scale-up vs scale-out
General AI-infrastructure concept · Concept · Current · Beginner
Scale-up joins accelerators into one tightly coupled memory-sharing domain; scale-out connects many domains over a data-centre network. An NVL72 rack is one scale-up domain; 100 such racks on InfiniBand form a scale-out cluster.
NVLink 5
NVIDIA, Blackwell generation · Interface · Current · Intermediate
Fifth-generation GPU-to-GPU link giving about 1.8 TB/s of bidirectional bandwidth per GPU, around 14 times PCIe Gen5 x16. All 72 GPUs in a GB200 NVL72 talk to each other over NVLink 5 at full speed.
NVLink 6
NVIDIA, Vera Rubin generation (2026) · Interface · Emerging · Intermediate
Sixth-generation NVLink doubling per-GPU bandwidth to about 3.6 TB/s. A Vera Rubin NVL72 rack totals roughly 260 TB/s of NVLink bandwidth.
NVSwitch
NVIDIA · Component · Current · Intermediate
Switch chip that connects many NVLink ports so every GPU can reach every other GPU at full bandwidth. NVL72 racks contain NVLink switch trays packed with NVSwitch chips.
NVLink Fusion
NVIDIA, announced May 2025 · Technology · Current · Advanced
Programme licensing NVLink so other companies' custom CPUs and accelerators can join NVIDIA's scale-up fabric. Partners such as Qualcomm and Fujitsu announced CPUs that connect to NVIDIA GPUs via NVLink Fusion.
UALink 1.0
UALink Consortium (AMD, Intel, Google, Microsoft, Meta, AWS and others), April 2025 · Standard · Emerging · Intermediate
Open scale-up interconnect standard at 200 Gb/s per lane for up to 1,024 accelerators in one pod. Vendors plan UALink switches so chips from different makers can share one scale-up domain.
Infinity Fabric GPU links (xGMI)
AMD · Interface · Current · Intermediate
AMD's GPU-to-GPU links, built on its Infinity Fabric interconnect, that fully connect the Instinct GPUs within one server. Eight MI300X GPUs in a server are fully connected by Infinity Fabric links.
InfiniBand
InfiniBand Trade Association; NVIDIA Quantum switches · Standard · Current · Intermediate
Low-latency, lossless network with native RDMA, long the default for supercomputers and AI clusters. NVIDIA Quantum-X800 switches provide 800 Gb/s InfiniBand ports for large GPU clusters.
Ultra Ethernet (UEC 1.0)
Ultra Ethernet Consortium (Linux Foundation), June 2025 · Standard · Emerging · Intermediate
Specification modernising Ethernet transport for AI and HPC, with better congestion control and packet spraying. Network vendors ship Ultra Ethernet-ready NICs and switches as an open alternative to InfiniBand.
RoCE v2
InfiniBand Trade Association · Standard · Current · Intermediate
RDMA over Converged Ethernet: runs InfiniBand-style direct memory transfers over routable Ethernet/IP networks. Many AI clusters use RoCE v2 on 400 GbE switches instead of InfiniBand.
GPUDirect RDMA
NVIDIA (AMD offers equivalent with ROCm) · Technology · Current · Advanced
Lets network cards move data directly between GPU memories on different servers without copying through CPU memory. Gradient exchanges between nodes skip the CPU, cutting latency and freeing CPU cycles.
SuperNICs and DPUs
NVIDIA ConnectX and BlueField; AMD Pensando; others · Component · Current · Intermediate
High-speed network adapters, sometimes with their own processors, that offload networking and security from the CPU. Each GPU in a cluster pairs with an 800 Gb/s ConnectX-8 SuperNIC.
All-reduce collective
MPI and NCCL terminology · Concept · Current · Intermediate
Operation where every GPU contributes data and all receive the combined result, used to average gradients. After each training step, an all-reduce makes every GPU's gradients identical.
Rail-optimised topology
AI cluster network design · Architecture · Current · Advanced
Wiring where GPU number k in every server connects to the same 'rail' switch, shortening common communication paths. GPU 3 of all 128 servers shares one leaf switch, so their traffic needs just one hop.
NVIDIA Spectrum-X
NVIDIA (Spectrum switches with ConnectX and BlueField SuperNICs) · System · Current · Intermediate
NVIDIA's Ethernet platform for AI clusters, pairing Spectrum switches with SuperNICs for adaptive routing and congestion control. A cloud builder chooses Spectrum-X over InfiniBand so its AI cluster reuses existing Ethernet skills and tools.
Pluggable optics (800G and 1.6T)
OSFP and QSFP-DD transceivers from many vendors · Component · Current · Intermediate
Laser modules that turn electrical signals into light so links can span racks and halls; AI clusters need several per GPU. A 10,000-GPU cluster can need tens of thousands of 800G transceivers, a large share of network cost and power.