Distributed Training and AI Hardware
How large models are trained across thousands of accelerators: training compute and scaling laws, GPU architecture and the roofline model, TPUs and other accelerators, numerical precision, memory budgets, data parallelism and collectives, ZeRO and FSDP sharding, tensor, sequence and expert parallelism, pipeline and 3D parallelism, clusters, networks and storage, reliability and profiling, and the energy and economics of compute.
AI-480 Year 4 3 credits 12 lessons · 4 h 56 min
Offered by HSM University · Artificial Intelligence
This is the American version of this course. See the Canadian version.
What you'll learn
- Estimate training compute with 6ND, apply scaling laws and compute-optimal ratios, and convert compute into time using MFU
- Explain GPU execution, tensor cores and the memory hierarchy, and use the roofline model to classify kernels as memory- or compute-bound
- Describe systolic arrays, TPUs and other AI accelerators and compare platforms by measured cost and software maturity
- Explain floating-point formats and apply mixed-precision, loss scaling, FP8 scaling and inference quantization
- Build a memory budget for training and inference, including optimizer state, activations, recomputation and the KV cache
- Implement and analyze data parallelism and collective operations, including the ring all-reduce and the alpha-beta cost model
- Apply ZeRO and FSDP sharding stages and calculate their memory savings and communication costs
- Explain tensor, sequence, context and expert parallelism and where each belongs in a cluster
- Calculate pipeline bubbles for GPipe, 1F1B and interleaved schedules and design 3D parallel configurations
- Describe server, network and storage architectures for training clusters and map parallelism onto topologies
- Plan checkpointing and recovery, detect stragglers and silent data corruption, and profile training steps to raise MFU and goodput
- Estimate the energy, emissions and cost of training and inference and produce an integrated training plan
Course content
12 lessons · 4 h 56 min total · final exam
- 1 The scale of modern AI training 25 min · quiz
- 2 GPU architecture and the roofline model 25 min · quiz
- 3 TPUs, systolic arrays and other AI accelerators 25 min · quiz
- 4 Numerical precision and mixed-precision training 24 min · quiz
- 5 Where the memory goes: parameters, optimizer state and activations 24 min · quiz
- 6 Data parallelism and collective communication 25 min · quiz
- 7 Sharded data parallelism: ZeRO and FSDP 25 min · quiz
- 8 Tensor, sequence, context and expert parallelism 25 min · quiz
- 9 Pipeline parallelism and 3D parallelism 24 min · quiz
- 10 Clusters, interconnects and storage 24 min · quiz
- 11 Reliability, profiling and efficiency at scale 25 min · quiz
- 12 Energy, cost and the economics of compute 25 min · quiz
- E Final exam 20 questions
Requirements
Recommended background. You can start any course at any time.
Description
How large models are trained across thousands of accelerators: training compute and scaling laws, GPU architecture and the roofline model, TPUs and other accelerators, numerical precision, memory budgets, data parallelism and collectives, ZeRO and FSDP sharding, tensor, sequence and expert parallelism, pipeline and 3D parallelism, clusters, networks and storage, reliability and profiling, and the energy and economics of compute.
Part of the Science stream, typically taken in year 4.