Pipelines, orchestration and continuous training

A model that is retrained by hand, by a data scientist running notebook cells in the right order, will eventually be retrained wrongly or not at all. Production ML replaces manual steps with pipelines: automated, versioned sequences of steps that extract data, validate it, build features, train, evaluate and register models, run on a schedule or a trigger, and leave a complete record. This lesson covers how pipelines are structured as directed acyclic graphs, the orchestration tools that run them, the properties that make pipelines reliable, and how continuous integration, delivery and training fit together into an automated loop with the right human checkpoints.

Pipelines as graphs

An ML pipeline is naturally described as a directed acyclic graph (DAG): each node is a step, and each edge says that one step needs another's output. Training depends on the train-test split, which depends on features, which depend on validated data. The orchestrator computes an order that respects the dependencies (a topological sort), runs independent steps in parallel, retries failed steps, and records the inputs, outputs, logs and timing of each.

import hashlib
from graphlib import TopologicalSorter

# A training pipeline as a DAG: each step lists the steps it depends on
dag = {
    "extract": [],
    "validate": ["extract"],
    "features": ["validate"],
    "split": ["features"],
    "train": ["split"],
    "evaluate": ["train", "split"],
    "register": ["evaluate"],
}
code_version = {step: "v1" for step in dag}
cache = {}

def run_pipeline(data_version):
    order = list(TopologicalSorter(dag).static_order())
    keys, ran = {}, []
    for step in order:
        parts = [step, code_version[step]] + [keys[d] for d in dag[step]]
        if step == "extract":
            parts.append(data_version)
        keys[step] = hashlib.sha256("|".join(parts).encode()).hexdigest()[:10]
        if keys[step] not in cache:          # run only if inputs or code changed
            cache[keys[step]] = True
            ran.append(step)
    return order, ran

order, ran = run_pipeline("2026-10-01")
print("Execution order:", " -> ".join(order))
print("Run 1 (new data):       ran:", ", ".join(ran))
print("Run 2 (nothing changed): ran:", ", ".join(run_pipeline("2026-10-01")[1]) or "nothing")
code_version["train"] = "v2"                 # a new hyperparameter in the training step
print("Run 3 (train code v2):  ran:", ", ".join(run_pipeline("2026-10-01")[1]) or "nothing")
print("Run 4 (new data):       ran:", ", ".join(run_pipeline("2026-10-08")[1]))

Execution order: extract -> validate -> features -> split -> train -> evaluate -> register
Run 1 (new data):       ran: extract, validate, features, split, train, evaluate, register
Run 2 (nothing changed): ran: nothing
Run 3 (train code v2):  ran: train, evaluate, register
Run 4 (new data):       ran: extract, validate, features, split, train, evaluate, register

The example shows a powerful pattern: each step gets a key computed from its code version and the keys of its inputs (and, for the first step, the data version). If a step's key has been seen before, its cached output is reused. Rerunning with nothing changed does no work; changing only the training code reruns training and the steps after it, reusing the extracted data, features and split; new data reruns everything. Orchestration tools such as Kubeflow Pipelines, ZenML, Metaflow and Dagster provide this kind of caching, which saves both time and money and makes the lineage of every artifact explicit (Lesson 3).

Orchestration tools

Orchestrators fall into broad families. General workflow schedulers, led by Apache Airflow, run DAGs of tasks on schedules and are widely used for data pipelines; Prefect and Dagster are newer alternatives with stronger support for data assets and testing. ML-focused orchestrators such as Kubeflow Pipelines, Vertex AI Pipelines, SageMaker Pipelines, Metaflow and ZenML add artifact tracking, caching and integration with training infrastructure. Distributed computing frameworks such as Ray and Spark run the heavy computation inside steps. The choice matters less than the practices; many organizations use one scheduler for data pipelines and another for ML, connected by data availability triggers.

Properties of reliable pipelines

• Idempotency: running a step twice with the same inputs produces the same result and no side effects such as duplicated rows. This makes retries safe.
• Parameterization by time: steps take the data interval they process as a parameter (for example, "the week ending 8 October") rather than reading "now", so any past run can be reproduced and missing periods can be backfilled.
• Atomic outputs: a step writes its result to a temporary location and publishes it only when complete, so downstream steps never read half-written data.
• Validation gates: data validation (Lesson 2) and model evaluation (Lesson 4) are steps that can stop the pipeline, not reports someone may read later.
• Observability: each run reports its status, duration and key metrics; failures alert an owner.
• Isolation: each step runs in its own versioned container, so dependency changes in one step do not break others.

CI, CD and CT

MLOps extends software's continuous integration and continuous delivery with a third loop:

• Continuous integration (CI): every change to pipeline code, feature code or model code triggers automated tests: unit tests, data and schema tests, a quick training run on a small sample, and golden serving tests (Lesson 4).
• Continuous delivery (CD): a change that passes CI is packaged and deployed as a new version of the pipeline or serving system, through the deployment strategies of Lesson 5.
• Continuous training (CT): the deployed pipeline retrains models automatically on schedules or triggers (Lesson 7), producing candidate models that pass through evaluation gates and, if they qualify, through the release process.

Two different things are therefore being versioned and released: the pipeline (code) and the models it produces (artifacts). A bug fix to feature code goes through CI/CD; a weekly retrain on new data goes through CT. Both end with gates before anything reaches users.

Champion and challenger

In an automated loop, the current production model is the champion and each retrained or newly developed model is a challenger. A challenger replaces the champion only if it wins on predefined criteria: offline metrics on recent data, slice metrics, behavioural tests, cost and latency, and where appropriate an online experiment (Lesson 8). Keeping the champion in place unless the challenger clearly wins protects against regressions from bad data or unlucky training runs. For high-impact models, a human approval step remains before promotion; for many routine models, full automation with good gates and monitoring is safer than a tired reviewer clicking approve.

Backfills and reproducibility

When a feature definition changes or a bug is fixed, historical feature values may need to be recomputed: a backfill. Because steps are parameterized by time and idempotent, a backfill is a matter of running the pipeline for each past interval, often in parallel. The same properties allow any past model to be reproduced: rerun the pipeline at the recorded code and data versions. Without them, recovering from a data bug can take weeks of manual work.

Infrastructure as code

The pipelines, schedules, compute resources, permissions and serving endpoints are themselves configuration that should be versioned and reviewed. Infrastructure as code tools (Terraform, Pulumi, Kubernetes manifests, Helm charts) describe the desired state in files kept in Git, and automated processes apply changes. This makes environments reproducible (a staging environment identical to production), changes reviewable, and disasters recoverable, because the whole system can be rebuilt from the repository and the versioned data.

Testing pipelines

Pipelines are code and deserve the same testing as any other software. Each step should have unit tests for its logic with small, hand-made inputs. Integration tests run the whole pipeline on a small, fixed sample of data in a test environment, checking that steps connect, that outputs have the expected schema and that the trained model reaches a sanity threshold. Contract tests check that the outputs of one team's pipeline match what the next team's pipeline expects. Because full training runs are slow and expensive, continuous integration usually runs these fast tests on every change and reserves full-scale runs for scheduled or release builds.

A useful technique is a staging environment that mirrors production with a subset of data. Pipeline changes are deployed there first, run for a few cycles, and compared with production outputs before promotion. This catches problems such as permissions, resource limits and scheduling conflicts that unit tests cannot.

Events and triggers

Schedules are simple, but data does not always arrive on schedule. Event-driven pipelines start when their inputs are ready: when an upstream table is marked complete, when a file lands in storage, or when monitoring raises a drift alert. Orchestrators support sensors or asset-based scheduling for this purpose, so that a training pipeline waits for validated data rather than running on stale or partial inputs at a fixed hour. Combining both, a schedule that also waits for its inputs, gives predictable timing with protection against incomplete data. Whatever the trigger, each run records why it started, which helps when investigating unexpected retraining.

Finally, pipelines need owners and on-call responsibility like any production service. A failed nightly retraining job that nobody notices for a month is a common way for models to drift into poor performance; alerts on pipeline failures and on the age of the production model close that gap.

Pipeline metadata, such as which runs produced which artifacts, with which inputs and parameters, should be queryable, not buried in logs. Questions such as "which models were trained on data from the faulty source last month?" then take minutes to answer.

Retention policies complete the picture. Pipelines produce large volumes of intermediate data and artifacts; keeping all of it forever is expensive, but deleting it too early breaks reproducibility. A typical policy keeps the artifacts of production models and their training data versions for as long as audit and regulation require, keeps experimental artifacts for a few months, and lets caches expire automatically. Writing the policy down, and enforcing it in the orchestration layer, avoids both runaway storage bills and the discovery, during an audit, that the data behind a production model no longer exists.