Machine Learning Systems and MLOps

Putting models into production: data pipelines, experiment tracking, testing, deployment, serving, monitoring, experimentation, cost and governance of ML systems.

AI-420 Year 4 3 credits 12 lessons · 4 h 51 min

Offered by HSM University · Artificial Intelligence

This is the American version of this course. See the Canadian version.

What you'll learn

Describe the components of a production ML system, why ML systems fail silently, and the MLOps maturity levels
Design data pipelines with validation, versioning and point-in-time correct features, and decide when a feature store is worthwhile
Track experiments reproducibly, compare models fairly under noise, and manage models through a registry with model cards
Test ML systems at the code, data, model, behavior and serving levels, and build automated release gates
Package models securely and release them with blue-green, canary and shadow deployments, with statistical canary analysis and rollback
Meet latency SLOs by applying percentiles, queueing theory, Little's law, batching, autoscaling and caching
Monitor service health, data, predictions and outcomes, detect covariate, prior and concept drift, and plan retraining
Design and analyze online experiments with correct sample sizes, trust checks, variance reduction and no peeking
Estimate and reduce the cost of training and serving using cost per prediction, purchasing options and shared clusters
Build idempotent, cached pipelines and combine continuous integration, delivery and training with champion-challenger gates
Apply SLOs and error budgets, defend against ML-specific security threats, monitor fairness, and govern models by risk tier
Design an end-to-end ML system whose framing, tests, release, monitoring, experiments, cost and governance fit together

Course content

12 lessons · 4 h 51 min total · final exam

1 From notebook to production 25 min · quiz
2 Data pipelines, validation and feature stores 25 min · quiz
3 Experiment tracking, reproducibility and model registries 25 min · quiz
4 Testing machine learning systems 24 min · quiz
5 Packaging and deployment patterns 24 min · quiz
6 Model serving, latency and throughput 24 min · quiz
7 Monitoring, drift and retraining 24 min · quiz
8 Online experiments and A/B testing 24 min · quiz
9 Cost and infrastructure 23 min · quiz
10 Pipelines, orchestration and continuous training 25 min · quiz
11 Reliability, security and governance 24 min · quiz
12 Case study and capstone project 24 min · quiz
E Final exam 20 questions

Requirements

Recommended background. You can start any course at any time.

Description

Putting models into production: data pipelines, experiment tracking, testing, deployment, serving, monitoring, experimentation, cost and governance of ML systems.

Part of the Science stream, typically taken in year 4.