Data pipelines, validation and feature stores

Every machine learning system is fed by data pipelines, and most production ML incidents start in them: a source system changes a column's meaning, a job runs twice, an upstream table arrives late, or a feature is computed with information that would not have been available at prediction time. This lesson covers how data flows from sources to training sets and to serving, how data are versioned, how pipelines validate what they receive, the subtle problem of point-in-time correctness, and the feature stores that many organizations build to compute features once and use them consistently in training and serving.

From sources to features

Data for ML typically comes from operational databases (orders, accounts, payments), event streams (clicks, sensor readings, logs), third-party sources and human labeling. Pipelines extract these data, transform them, and load them into storage for analysis and training: a data warehouse for structured tables, a data lake for raw files, or a lakehouse that combines both. Transformations clean records, join sources, and compute features: aggregates such as the number of purchases in the last 30 days, encodings of categories, text embeddings, and so on.

Two pipeline styles are common. Batch pipelines run on a schedule, processing a day's or an hour's data at a time with tools such as SQL in a warehouse, Spark or dbt, orchestrated by schedulers such as Airflow (Lesson 10). Streaming pipelines process events continuously with systems such as Kafka and Flink, keeping features like "payments in the last ten minutes" up to date within seconds. Streaming is more complex and costly to build and operate, so it is reserved for features where freshness really matters.

Labels

Supervised models need labels, and labels are often the hardest data to get. Some come naturally from later events: whether a customer churned, whether a payment was disputed, how many units sold. These natural labels arrive with a delay, sometimes weeks or months (a chargeback can arrive 90 days after a payment), which affects both training and monitoring (Lesson 7). Other labels require human annotation, with its own pipeline of guidelines, quality checks, agreement measures and cost. Whatever the source, the label definition must be written down precisely, because subtle changes, such as counting a churned customer from the date of cancellation rather than the date of the last order, change what the model learns.

Versioning data

To reproduce a model, you need the exact data it was trained on, not just the code. Data versioning tools record which version of each dataset a training run used. Approaches include immutable, dated snapshots of tables; table formats such as Delta Lake, Apache Iceberg and Apache Hudi that support time travel (querying a table as it was at a past time); and tools such as DVC that track large files alongside code in Git. A good rule is that training reads only from versioned, immutable inputs, and the version identifiers are logged with the run (Lesson 3).

Validating data

Because bad data rarely causes errors, pipelines must check data explicitly before using it. Data validation compares each new batch against expectations:

• Schema: expected columns, types and allowed values; no unexpected new categories.
• Completeness: row counts within an expected range; missing value rates below thresholds.
• Ranges and constraints: values within plausible limits (ages between 0 and 120, prices positive); unique keys unique.
• Distributions: summary statistics similar to recent batches, measured with distance metrics such as the population stability index (Lesson 7).
• Freshness: the newest record recent enough, so a stalled upstream job is caught.
• Relationships: totals that should reconcile with source systems.

Libraries such as Great Expectations, TensorFlow Data Validation and pandera let teams declare these expectations as code, run them in the pipeline, and stop or quarantine a batch that fails. The response to a failure should be planned: block training on a bad batch, alert the owner, and fall back to the last good data.

Point-in-time correctness and leakage

Data leakage occurs when a model is trained with information that will not be available when it makes predictions. A subtle and common form arises when features are joined to labels. Suppose a credit model predicts on day 45 whether a customer will default by day 55. Each training example must use the feature values as they were on day 45. If the pipeline simply joins each customer to the latest value of each feature, it will use values from after day 45, which may reflect the very default being predicted. The offline evaluation then looks excellent and the production model disappoints.

import numpy as np
import pandas as pd

# Point-in-time correctness: join each label to the feature value known at prediction time
rng = np.random.default_rng(1)
n_users = 5000
base = rng.uniform(100, 1000, n_users)
shock = rng.random(n_users) < 0.2                     # 20 % of users lose income on day 50
rows = []
for d in range(0, 60, 5):                              # balance snapshot every 5 days
    bal = base * rng.uniform(0.85, 1.15, n_users) - 450 * shock * (d >= 50)
    rows += [(u, d, b) for u, b in zip(range(n_users), bal)]
feats = pd.DataFrame(rows, columns=["user", "day", "balance"])

# Prediction is made on day 45; the label is whether the balance is below 150 on day 55
labels = pd.DataFrame({"user": np.arange(n_users), "day": 45})
labels["default"] = (feats[feats.day == 55].balance.values < 150).astype(int)

latest = feats[feats.day == 55][["user", "balance"]]
leaky = labels.merge(latest, on="user")                                  # day-55 data: the future
asof = pd.merge_asof(labels.sort_values("day"), feats.sort_values("day"),
                     on="day", by="user", direction="backward").sort_values("user")

def accuracy(df):
    return ((df.balance < 150).astype(int).values == df["default"].values).mean()

print(f"Default rate: {labels.default.mean():.3f}")
print(f"Offline accuracy, latest feature values (leaky): {accuracy(leaky):.3f}")
print(f"Offline accuracy, point-in-time join:            {accuracy(asof):.3f}")
print(f"Always predicting no default:                   {1 - labels.default.mean():.3f}")

Default rate: 0.155
Offline accuracy, latest feature values (leaky): 1.000
Offline accuracy, point-in-time join:            0.889
Always predicting no default:                   0.845

The example is deliberately extreme: the label is defined from the day-55 balance, so the leaky join, which uses day-55 balances, scores a perfect 100 % offline. The point-in-time join, implemented with pandas merge_asof so that each label sees only the latest snapshot at or before day 45, scores 88.9 %, only modestly better than the 84.5 % achieved by always predicting no default. The honest number is the one production will deliver. Real leakage is rarely this blatant, but it follows the same pattern: offline results that seem too good, and features whose values were updated after the event being predicted. The defense is to build training sets with point-in-time joins (sometimes called as-of joins) keyed on the prediction time, and to be suspicious of any feature that is much more predictive than domain experts expect.

Feature stores

As organizations build more models, the same features get recomputed by different teams in different ways, and training-serving skew creeps in (Lesson 1). A feature store is a system for defining features once and serving them consistently. Its typical components are:

• A feature registry: definitions, owners, documentation and lineage of each feature, so teams can discover and reuse them.
• An offline store: historical feature values, usually in a warehouse or lake, supporting point-in-time joins to build training sets.
• An online store: the latest feature values in a low-latency database such as Redis or DynamoDB, read by the serving system in milliseconds.
• Materialization jobs: batch or streaming pipelines that compute features from the same definitions and write them to both stores.

Open-source Feast and commercial products from cloud providers and specialist vendors implement this pattern. A feature store is not always worth its cost: a team with a few batch models may do well with shared SQL and careful tests. But for organizations with many online models, it is often the most effective defense against skew and duplicated effort.

Data quality is an ongoing job

Data pipelines need owners, service level objectives (for example, the daily table available by 06:00 with 99.5 % reliability), monitoring, and on-call responses like any production service. Contracts between data producers and consumers, sometimes formalized as data contracts, describe the schema, meaning and quality guarantees of each dataset, so that a team changing its database knows who depends on it. Lineage tools record which tables and jobs feed which features and models, so that when a source breaks, the affected models can be identified quickly.

In the United States, personal information in training data is governed by a patchwork of sector laws (such as HIPAA for health data and GLBA for financial data) and state privacy laws such as the California Consumer Privacy Act as amended by the CPRA, which give consumers rights over their data and, in some states, over automated decision-making.

Privacy rules affect pipeline design directly: data minimization, retention limits, deletion requests that must propagate to training sets, and access controls on sensitive features (Lesson 11).

Backfills and late data

Real pipelines must also cope with data that arrives late or is corrected after the fact. Events from mobile devices may reach the server hours after they happened; a payment may be reversed days later; a source system may restate last month's figures. A pipeline that processes "yesterday's data" once and never again will silently miss these records. Robust designs partition data by event time rather than arrival time, reprocess recent partitions for a defined window (for example the last three days) on every run, and record when each partition was last updated, so that features and training sets can be rebuilt consistently. The same machinery supports backfills when a feature definition changes (Lesson 10).