Deep Learning
The core deep learning course, with every mechanism implemented from scratch in NumPy on small, seeded data: feedforward networks, activation functions and why depth helps, backpropagation as reverse-mode automatic differentiation with a working autodiff engine and gradient checks, stable losses, SGD, momentum, Adam and AdamW with warmup and cosine schedules, initialization, batch, layer and RMS normalization and residual networks, weight decay, dropout, augmentation, early stopping, memorization and double descent, convolutional networks and their size and receptive-field arithmetic, recurrent networks, BPTT, LSTMs and GRUs, scaled dot-product and multi-head attention, positional encodings and RoPE, the transformer block and the KV cache, transfer learning, contrastive pretraining, LoRA and scaling laws, autoencoders, VAEs, GANs and diffusion models, mixed precision, memory budgets, attribution, adversarial robustness and distribution shift, and a capstone project with calibration, energy costs, data provenance and Canadian and US governance.
CMPUT-466 Year 4 3 credits 12 lessons · 7 h 3 min
Offered by HSM University · Computing Science
This is the American version of this course. See the Canadian version.
What you'll learn
- Describe feedforward networks as compositions of affine maps and nonlinearities, compare activation functions, and explain what universal approximation and depth-efficiency results do and do not guarantee.
- Derive backpropagation as reverse-mode automatic differentiation on a computational graph, implement a small autodiff engine, and verify gradients with finite differences.
- Choose and justify loss functions and output layers, and compute softmax and cross-entropy stably with the log-sum-exp trick.
- Explain and apply SGD, momentum, Adam and AdamW, learning rate warmup and decay schedules, batch size scaling and gradient clipping.
- Use initialization, batch, layer and RMS normalization, and residual connections to keep signals and gradients well scaled in very deep networks.
- Apply weight decay, dropout, data augmentation and early stopping, and discuss memorization, double descent and implicit regularization.
- Compute convolution output sizes, receptive fields and parameter counts, and trace the main convolutional architectures from LeNet to efficient modern designs.
- Explain recurrent networks, backpropagation through time, vanishing and exploding gradients, and the gating of LSTMs and GRUs.
- Derive scaled dot-product and multi-head attention, implement causal self-attention, explain positional encodings including RoPE, and analyze transformer costs and the KV cache.
- Apply transfer learning, self-supervised and contrastive pretraining, and parameter-efficient fine-tuning, and reason with scaling laws and compute-optimal allocation.
- Compare autoencoders, VAEs, GANs and diffusion models by their objectives, sampling procedures and trade-offs.
- Debug and train networks reliably with mixed precision and memory budgets, evaluate attributions, robustness and distribution shift, and carry out a documented end-to-end project with attention to energy, data provenance and Canadian and US governance.
Course content
12 lessons · 7 h 3 min total · final exam
- 1 From linear models to deep networks 35 min · quiz
- 2 Backpropagation and automatic differentiation 36 min · quiz
- 3 Optimization for deep learning 35 min · quiz
- 4 Initialization, normalization and residual networks 34 min · quiz
- 5 Regularization and generalization in deep networks 36 min · quiz
- 6 Convolutional networks 32 min · quiz
- 7 Recurrent networks and sequence modeling 34 min · quiz
- 8 Attention and transformers 35 min · quiz
- 9 Pretraining, transfer and scaling 37 min · quiz
- 10 Generative models: autoencoders, VAEs, GANs and diffusion 36 min · quiz
- 11 Training in practice, interpretability and robustness 38 min · quiz
- 12 Capstone: responsible deep learning and an end-to-end project 35 min · quiz
- E Final exam 20 questions
Requirements
Recommended background. You can start any course at any time.
Description
The core deep learning course, with every mechanism implemented from scratch in NumPy on small, seeded data: feedforward networks, activation functions and why depth helps, backpropagation as reverse-mode automatic differentiation with a working autodiff engine and gradient checks, stable losses, SGD, momentum, Adam and AdamW with warmup and cosine schedules, initialization, batch, layer and RMS normalization and residual networks, weight decay, dropout, augmentation, early stopping, memorization and double descent, convolutional networks and their size and receptive-field arithmetic, recurrent networks, BPTT, LSTMs and GRUs, scaled dot-product and multi-head attention, positional encodings and RoPE, the transformer block and the KV cache, transfer learning, contrastive pretraining, LoRA and scaling laws, autoencoders, VAEs, GANs and diffusion models, mixed precision, memory budgets, attribution, adversarial robustness and distribution shift, and a capstone project with calibration, energy costs, data provenance and Canadian and US governance.
Part of the Science stream, typically taken in year 4.