Continuous integration, delivery and DevOps

Writing and testing code is only part of delivering software. The change must be combined with everyone else's work, built, checked, packaged, deployed to servers or app stores, watched in production, and rolled back if something goes wrong. When these steps are manual and infrequent, each release is large, risky and stressful; teams that release quarterly often spend weeks integrating and stabilizing. Continuous integration, continuous delivery and DevOps practices automate the path from commit to production so that releases become small, frequent and routine. This lesson covers continuous integration and build pipelines, continuous delivery and deployment strategies, feature flags, infrastructure as code, monitoring and incident response, the DORA metrics, and software supply-chain security. Lesson 2 introduced DevOps as a process; here are its practices.

Continuous integration

Continuous integration (CI), named by Grady Booch and made a core practice by Extreme Programming, means that every developer integrates their work into the shared main branch frequently, at least daily, and each integration is verified by an automated build that runs the tests. The point is to find integration problems within minutes, when the change is small and fresh in its author's mind, rather than weeks later. Martin Fowler's description lists the essential practices: keep a single source repository, automate the build, make the build self-testing, everyone commits to main every day, every commit triggers a build on an integration machine, fix broken builds immediately, keep the build fast, and make it easy for anyone to get the latest executable.

A CI server such as GitHub Actions, GitLab CI, Jenkins or Buildkite watches the repository and runs a pipeline on every push and pull request. A pipeline is a sequence of stages, each of which must pass before the next runs: typically checking out the code, installing dependencies, linting and formatting checks, type checking, unit tests, building artifacts, integration tests and security scans. Branch protection rules then prevent merging a pull request unless the pipeline passes. The most important cultural rule is that a broken main branch is everyone's top priority: if it stays red, people stop trusting it, merge on top of failures, and the benefit disappears.

Pipelines order stages from fast and cheap to slow and expensive, so that most problems are caught in the first minute. A common target is that the commit stage, which developers wait for, finishes within about ten minutes. The configuration lives in the repository as code, so it is versioned and reviewed like everything else. The first listing below is a GitHub Actions workflow; it cannot be run here, because it runs on GitHub's servers. The second is the same checks as a local script, which really was run against a small project, so developers can reproduce a CI failure on their own machines.

# .github/workflows/ci.yml  (GitHub Actions; runs on GitHub's servers, shown as a listing)
name: ci
on:
  push:
    branches: [main]
  pull_request:
jobs:
  check:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.12", "3.13"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip
      - run: pip install -r requirements-dev.txt
      - run: ruff check --select F,E,B .
      - run: ruff format --check .
      - run: mypy --strict .
      - run: pytest -q --junitxml=results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results-${{ matrix.python-version }}
          path: results.xml

cat ci.sh
bash ci.sh

$ cat ci.sh
#!/usr/bin/env bash
# ci.sh: the same checks the CI server runs, so developers can run them locally
set -euo pipefail
echo "== lint";        ruff check --select F,E,B .
echo "== format";      ruff format --check .
echo "== types";       mypy --strict --no-error-summary .
echo "== tests";       pytest -q
echo "all checks passed"
$ bash ci.sh
== lint
All checks passed!
== format
2 files already formatted
== types
== tests
.                                                                        [100%]
1 passed in 0.01s
all checks passed

bash ci.sh

$ bash ci.sh
== lint
F841 Local variable `unused` is assigned to but never used
  --> shifts.py:13:9
   |
11 |             self.confirmed += 1
12 |             return "confirmed"
13 |         unused = 1
   |         ^^^^^^
14 |         return "waitlisted"
   |
help: Remove assignment to unused variable `unused`
Found 1 error.
No fixes available (1 hidden fix can be enabled with the `--unsafe-fixes` option).

The second run shows a principle worth copying: the script stops at the first failing stage (set -e), reports exactly what failed, and never runs the slower stages on code that is already known to be wrong. In a real pipeline, the developer sees the same message in the pull request within a minute or two.

Builds and artifacts

A build turns source code into something deployable: a Python wheel, a container image, a compiled binary, a mobile app bundle. Good builds are automated (one command), repeatable (the same inputs give the same outputs, which requires pinning dependency versions in a lock file), and fast (caching dependencies and test results). The output of the build is an artifact, which should be built once and promoted unchanged through test, staging and production environments, configured for each by environment variables rather than rebuilt. Rebuilding for production risks deploying something different from what was tested. Container images, usually built from a Dockerfile and stored in a registry, have become the standard artifact for server software because they package the application with its operating-system dependencies.

Continuous delivery and continuous deployment

Continuous delivery, described by Jez Humble and David Farley in their 2010 book of that name, extends CI so that every change that passes the pipeline is releasable: it has been built, tested in a production-like environment and could be deployed at the push of a button. Continuous deployment goes one step further and deploys every passing change to production automatically, with no manual approval. Continuous delivery suits organizations that need a business decision or regulatory approval for each release, such as app store submissions or medical software; continuous deployment suits web services where small, frequent releases are low risk. Both depend on a reliable automated test suite, because the pipeline, not a release manager, decides what is safe.

Canary releases, named after the canaries miners once carried to detect gas, deserve emphasis after lesson 1's CrowdStrike example: sending a new version to a few percent of users first, and comparing its error rate and latency with the old version, limits how many people a bad release can hurt. The comparison should be statistical rather than by eye, because small samples are noisy.

import math

def canary_check(base_req, base_err, can_req, can_err, z_limit=3.0):
    """Two-proportion z-test: is the canary's error rate significantly higher?"""
    p1, p2 = base_err / base_req, can_err / can_req
    pooled = (base_err + can_err) / (base_req + can_req)
    se = math.sqrt(pooled * (1 - pooled) * (1 / base_req + 1 / can_req))
    z = (p2 - p1) / se if se else 0.0
    verdict = "ROLL BACK" if z > z_limit else "continue"
    return f"baseline {p1:.3%}, canary {p2:.3%}, z = {z:5.2f} -> {verdict}"

# 95% of traffic on the old version, 5% on the canary, over the same 10 minutes
print(canary_check(base_req=95_000, base_err=190, can_req=5_000, can_err=12))
print(canary_check(base_req=95_000, base_err=190, can_req=5_000, can_err=31))

baseline 0.200%, canary 0.240%, z =  0.61 -> continue
baseline 0.200%, canary 0.620%, z =  6.16 -> ROLL BACK

Feature flags

A feature flag (or toggle) is a condition in the code that turns a feature on or off at run time, controlled by configuration rather than by a new deployment. Flags separate deploying code from releasing a feature: unfinished work can be merged to main and deployed while switched off, which makes trunk-based development (lesson 8) possible; a feature can be released gradually to a percentage of users, to a pilot group such as one food bank location, or to staff first; and a misbehaving feature can be switched off in seconds. Percentage rollouts should be sticky, so each user sees the same choice every time, which is usually achieved by hashing the user's identifier.

import hashlib

FLAGS = {"shift_swaps": {"enabled": True, "percent": 20, "allow": {"coordinator@example.org"}}}

def is_enabled(flag, user_id):
    cfg = FLAGS.get(flag)
    if not cfg or not cfg["enabled"]:
        return False
    if user_id in cfg["allow"]:
        return True
    digest = hashlib.sha256(f"{flag}:{user_id}".encode()).hexdigest()
    bucket = int(digest[:8], 16) % 100          # a stable bucket from 0 to 99 for each user
    return bucket < cfg["percent"]

users = [f"volunteer{i}@example.org" for i in range(1000)]
on = sum(is_enabled("shift_swaps", u) for u in users)
print(f"{on} of {len(users)} users see shift swaps (target 20%)")
print("same answer every time:", all(is_enabled("shift_swaps", u) == is_enabled("shift_swaps", u)
                                     for u in users))
print("coordinator always sees it:", is_enabled("shift_swaps", "coordinator@example.org"))

184 of 1000 users see shift swaps (target 20%)
same answer every time: True
coordinator always sees it: True

Flags carry a cost: every flag doubles the paths through the affected code, and old flags left in place become technical debt and occasionally cause incidents. The Knight Capital loss in lesson 1 involved an old flag reused for a new purpose. Teams therefore give each flag an owner and an expiry date, and remove release flags once a feature is fully rolled out.

Infrastructure as code

Infrastructure as code (IaC) describes servers, networks, databases, load balancers and their configuration in version-controlled files, applied by tools such as Terraform or OpenTofu, AWS CloudFormation, Pulumi, Ansible and, for containers, Kubernetes manifests. Instead of an administrator clicking through a console, the desired state is written down, reviewed in pull requests, tested and applied automatically, so environments can be recreated exactly and differences between staging and production disappear. The same discipline applies to configuration: the twelve-factor app methodology recommends keeping configuration in environment variables, separate from code, and keeping development, staging and production as similar as possible.

Monitoring, observability and incidents

Once software is in production, the team needs to know whether it is working. Monitoring watches known indicators and alerts when they cross thresholds. Observability is the broader ability to understand a system's internal state from its outputs, especially to answer questions nobody anticipated. It rests on three kinds of telemetry: metrics (numbers over time, such as request rate, error rate and latency percentiles), logs (timestamped records of events, best written as structured key–value data) and traces (the path of one request through many services). OpenTelemetry has become the standard for collecting all three. Google's site reliability engineering (SRE) practice recommends watching four golden signals for every service: latency, traffic, errors and saturation.

SRE also introduced service level objectives and error budgets. A service level indicator (SLI) is a measured quantity, such as the proportion of requests that succeed within 300 milliseconds. A service level objective (SLO) is a target for it, such as 99.9 percent over 30 days. The error budget is what the SLO allows to fail: 0.1 percent of requests, or about 43 minutes of full downtime in 30 days. While budget remains, the team ships features freely; when it is spent, the team slows releases and works on reliability. The budget turns the old argument between speed and stability into an agreed, measurable rule.

# Error budget for a 30-day SLO, and how much of it has been spent
slo = 0.999
days = 30
total_requests = 4_200_000
failed_requests = 2_730
budget = (1 - slo) * total_requests
minutes = (1 - slo) * days * 24 * 60
print(f"SLO {slo:.1%}: budget of {budget:,.0f} failed requests "
      f"(about {minutes:.0f} minutes of total downtime) per {days} days")
spent = failed_requests / budget
print(f"failed so far: {failed_requests:,} = {spent:.0%} of the budget")
print("policy:", "freeze risky releases, prioritize reliability" if spent >= 1
      else "slow down and review recent changes" if spent >= 0.5
      else "ship normally")

SLO 99.9%: budget of 4,200 failed requests (about 43 minutes of total downtime) per 30 days
failed so far: 2,730 = 65% of the budget
policy: slow down and review recent changes

When something does go wrong, a clear incident process helps: someone takes the incident commander role and coordinates, others investigate, a communications lead keeps users informed, and the first priority is to restore service, often by rolling back or switching off a feature flag, before diagnosing the root cause. Afterwards, a blameless post-incident review asks how the system and process allowed the failure and what will prevent it, rather than who made the mistake. Blame teaches people to hide problems; blameless reviews, standard practice at Google, Etsy and many others, surface the latent conditions that cause the next incident.

Measuring delivery: the DORA metrics

The DevOps Research and Assessment (DORA) program, founded by Nicole Forsgren, Jez Humble and Gene Kim and described in the 2018 book Accelerate, identified four key metrics of software delivery performance: deployment frequency (how often the team deploys to production), lead time for changes (from commit to running in production), change failure rate (the percentage of deployments that cause a failure needing remediation), and time to restore service after a failure. In 2024 DORA added a fifth, rework rate (the share of deployments that are unplanned fixes), and grouped the metrics into throughput and instability measures. The 2025 report, renamed The State of AI-assisted Software Development, replaced its familiar low, medium, high and elite clusters with team archetypes based on a wider set of measures, including product performance, friction and burnout. The enduring finding is that throughput and stability go together: the teams that deploy most often also fail least and recover fastest, because small, automated, well-tested changes are both quicker and safer.

from datetime import datetime as dt, timedelta
from statistics import median

# One month of production deployments: (deployed_at, oldest commit included, caused_failure, restored_at)
deploys = [
    (dt(2026, 10, 1, 10), dt(2026, 9, 30, 15), False, None),
    (dt(2026, 10, 2, 14), dt(2026, 10, 2, 9), False, None),
    (dt(2026, 10, 5, 11), dt(2026, 10, 2, 16), True, dt(2026, 10, 5, 11, 40)),
    (dt(2026, 10, 7, 9), dt(2026, 10, 6, 13), False, None),
    (dt(2026, 10, 9, 16), dt(2026, 10, 8, 10), False, None),
    (dt(2026, 10, 14, 10), dt(2026, 10, 9, 17), False, None),
    (dt(2026, 10, 15, 15), dt(2026, 10, 15, 11), False, None),
    (dt(2026, 10, 20, 13), dt(2026, 10, 16, 9), True, dt(2026, 10, 20, 15, 5)),
    (dt(2026, 10, 22, 10), dt(2026, 10, 21, 14), False, None),
    (dt(2026, 10, 27, 11), dt(2026, 10, 23, 15), False, None),
]
weeks = 4
lead = [(d - c) / timedelta(hours=1) for d, c, _, _ in deploys]
fails = [(r - d) / timedelta(minutes=1) for d, _, f, r in deploys if f]
print(f"deployment frequency : {len(deploys) / weeks:.1f} per week")
print(f"lead time for changes: median {median(lead):.0f} hours")
print(f"change failure rate  : {len(fails) / len(deploys):.0%}")
print(f"time to restore      : median {median(fails):.0f} minutes")

deployment frequency : 2.5 per week
lead time for changes: median 25 hours
change failure rate  : 20%
time to restore      : median 82 minutes

Like velocity in lesson 4, these metrics are for a team to understand and improve its own system, not for ranking teams or individuals. Used as targets, they invite gaming: splitting deployments to raise frequency, or classifying failures to lower the failure rate.

Software supply-chain security

Modern applications are mostly other people's code: a typical web application depends directly and indirectly on hundreds of open-source packages, and its build runs on third-party servers using third-party actions and images. Attackers have noticed. The 2020 SolarWinds attack inserted malicious code into a vendor's build system and reached thousands of customers through a signed update; the 2021 Log4Shell vulnerability in the ubiquitous Log4j library forced emergency patching worldwide; and the 2024 XZ Utils backdoor, planted over two years by an attacker who gained a maintainer's trust, was caught by chance days before it would have shipped in major Linux distributions. Defenses include pinning dependency versions in lock files and verifying their hashes, scanning dependencies for known vulnerabilities (Dependabot, pip-audit, OSV-Scanner) and keeping them updated, generating a software bill of materials (SBOM) in a standard format such as SPDX or CycloneDX that lists every component, signing artifacts (for example with Sigstore), and hardening the build itself, as described by the SLSA (Supply-chain Levels for Software Artifacts) framework.

Regulation is following. The European Union's Cyber Resilience Act applies to software sold in the EU, including by American companies: manufacturers must report actively exploited vulnerabilities and severe incidents from 11 September 2026, and most other obligations, including secure-by-design requirements, vulnerability handling and documentation of components, apply from 11 December 2027. In the United States, Executive Order 14028 of 2021 led federal agencies to require software suppliers to attest that they follow the NIST Secure Software Development Framework (SSDF) and to encourage SBOMs, and CISA's Secure by Design initiative asks vendors to take ownership of customers' security outcomes. CISSP-113 and CMPUT-331 cover these requirements in more depth; check current federal guidance, which has been revised several times.