Optimization for Machine Learning

What you'll learn

• Analyze gradient descent using smoothness, strong convexity and the condition number, and choose step sizes and preconditioning
• Test convexity, compute smoothness and strong convexity constants from data, use subgradients and apply optimality conditions including soft thresholding
• Explain stochastic gradient noise, batch size and learning rate schedules, and apply the linear scaling rule and its limits
• Apply Newton, BFGS, L-BFGS, Gauss-Newton and Levenberg-Marquardt methods, and explain Hessian-vector products and structured curvature such as K-FAC and Shampoo
• Implement and compare momentum, Nesterov acceleration, Adam and AdamW, and manage clipping and optimizer memory
• Explain forward- and reverse-mode automatic differentiation, gradient checkpointing, mixed-precision gradients and gradient checking
• Formulate convex problems, apply duality and the KKT conditions, and solve constrained and sparse problems with projected and proximal methods
• Relate optimization to generalization through regularization, early stopping, implicit bias, step size, flat minima and sharpness-aware minimization
• Describe non-convex landscapes, curvature dynamics, initialization, precision and scaling behavior in deep network training
• Tune hyperparameters with random search, Bayesian optimization and successive halving, and set learning rates and transfer them across scale
• Solve black-box and combinatorial problems with direct search, SPSA, CMA-ES, simulated annealing and mixed-integer programming
• Plan distributed and federated training, accounting for memory, sharding, communication and large-batch effects, and diagnose a training run