Splits, validation design and leakage

A model's reported performance is only as trustworthy as the way it was evaluated. The central rule of machine learning evaluation is simple: measure performance on data the model has never seen, drawn from the same conditions the model will face in use. Applying that rule correctly is surprisingly hard. Random splits can place nearly identical records in training and test sets; preprocessing fitted on all the data quietly leaks test information; features may contain information that would not be available at prediction time; and repeated tuning against a test set gradually overfits it. Leakage is one of the most common reasons why published and project results fail to hold up. This lesson covers training, validation and test splits, choosing split strategies for grouped, temporal and spatial data, cross-validation, the many forms of leakage, and protecting a final test set.

Training, validation and test sets

Data is usually divided into three parts. The training set is used to fit model parameters. The validation set, also called the development set, is used to compare models, choose hyperparameters, select features and make other design decisions. The test set is held back and used once, at the end, to estimate how the final model will perform on new data. The separation matters because every decision made by looking at a dataset fits the model a little more closely to that dataset; if the test set is used for decisions, its estimate becomes optimistic. Typical proportions are 70/15/15 or 80/10/10, but the right sizes depend on how much data is available and how precisely performance must be estimated (Lesson 8).

Choosing a split strategy

Random splitting assumes that each record is independent and that future data resembles the past. Many real datasets violate these assumptions, and the split must reflect how the model will be used:

• Grouped data: when several records belong to the same entity, such as multiple images of one patient, several transactions from one customer or many sentences from one document, all records from an entity must be in the same split. Otherwise the model can memorize entity-specific features and appear to generalize when it does not.
• Temporal data: when the model will predict the future, train on earlier data and test on later data, never the reverse. Random splits of time series let the model "see the future".
• Spatial data: for maps, satellite imagery and sensor networks, nearby locations are correlated, so splits by region give more honest estimates for new locations.
• New-entity generalization: if the system must work for new suppliers, new hospitals or new users, test on entities absent from training.
• Stratification: for rare classes, stratified splitting keeps class proportions similar across splits.

import numpy as np
# Leakage: random splits versus group splits when patients have several records
rng = np.random.default_rng(2)
patients, per = 300, 6
pid = np.repeat(np.arange(patients), per)
signature = rng.normal(0, 1, (patients, 8))        # each patient has a stable "fingerprint"
X = signature[pid] + rng.normal(0, 0.3, (patients * per, 8))
y = (rng.random(patients) < 0.5)[pid]               # label depends only on the patient, not on the features
def knn_accuracy(train, test):
    d = ((X[test][:, None, :] - X[train][None, :, :]) ** 2).sum(-1)
    nearest = train[np.argmin(d, axis=1)]
    return np.mean(y[nearest] == y[test])
idx = rng.permutation(len(y)); cut = int(0.8 * len(y))
print(f"random split accuracy: {knn_accuracy(idx[:cut], idx[cut:]):.2f}  (looks impressive)")
groups = rng.permutation(patients); test_p = groups[:60]
test = np.where(np.isin(pid, test_p))[0]; train = np.where(~np.isin(pid, test_p))[0]
print(f"group split accuracy:  {knn_accuracy(train, test):.2f}  (the honest answer: chance)")

random split accuracy: 0.94  (looks impressive)
group split accuracy:  0.46  (the honest answer: chance)

In this simulation, labels depend only on which patient a record belongs to, not on anything a model could generalize from. With a random split, a nearest-neighbour model finds other records of the same patient in the training set and appears 94 % accurate. With a group split, where test patients are never seen in training, accuracy falls to chance. Real projects show the same effect less dramatically: grouped leakage routinely inflates medical imaging, speech and customer analytics results.

Cross-validation

When data is limited, a single validation split gives a noisy estimate. K-fold cross-validation divides the training data into \(k\) folds, trains on \(k - 1\) of them and validates on the remaining fold, rotating so that each fold is used once for validation, and averages the results. Five or ten folds are common. Cross-validation must respect the same structure as the main split: grouped k-fold keeps entities together, and time-series cross-validation uses expanding or rolling windows that always validate on later data. When hyperparameters are tuned within cross-validation, nested cross-validation, with an inner loop for tuning and an outer loop for evaluation, gives an unbiased estimate. For large datasets and expensive models, a single well-designed validation split is usually sufficient.

Forms of leakage

Leakage occurs whenever information that would not be available at prediction time influences training or evaluation. Common forms include:

• Duplicate and near-duplicate leakage: the same or nearly the same records in training and test sets, such as reposted images, re-submitted forms or boilerplate documents.
• Group leakage: records from the same entity across splits, as above.
• Temporal leakage: features computed using future information, such as a customer's total annual spend used to predict whether they churn in March.
• Target leakage: features that are consequences or proxies of the label, such as a "treatment given" field when predicting diagnosis, or an "account closed" flag when predicting churn.
• Preprocessing leakage: normalization statistics, imputation values, vocabulary or feature selection computed on the whole dataset before splitting, which lets test data influence training.
• Evaluation leakage: repeatedly testing on the test set while developing, or choosing among many models by their test score, which turns the test set into a validation set.
• Benchmark contamination: for large language models, test questions that appeared in pre-training data, so that the model has memorized answers.

Pipelines and reproducible splits

The most reliable way to prevent preprocessing leakage is to encapsulate all preprocessing steps, such as imputation, scaling, encoding and feature selection, in a pipeline object that is fitted only on training data and then applied to validation and test data. Splits should be created once, with a recorded random seed and group or time rules, saved as lists of record identifiers, and reused by every experiment, so that all team members compare models on exactly the same data. Version the split files alongside the code.

Protecting the test set

The test set should be used as rarely as possible, ideally once for the final model and once more if a major error is found. Practical measures include storing it separately with restricted access, having one team member responsible for running final evaluations, recording every test evaluation in the decision log, and agreeing before looking at test results which model will be reported. If the test set has been used too often, or if the final system changed substantially after testing, report this honestly and, if possible, collect a fresh test set. For language model projects, private test sets written by the team, or collected after the model's training cutoff, avoid contamination.

Near-duplicates and how to find them

Exact duplicates are easy to find by comparing records or file hashes; near-duplicates are harder and often more common. Examples include the same photo resized or cropped, the same document saved in two formats, or forum posts that quote each other. Techniques for finding them include perceptual hashing for images, which produces similar hashes for visually similar pictures; MinHash and locality-sensitive hashing for text, which estimate overlap between documents efficiently; and nearest-neighbour search in embedding space, flagging pairs that are unusually close. Once found, near-duplicates should be grouped and kept within one split, just like records from the same entity.

Validation sets for LLM projects

Projects built on large language models need validation and test sets too, even when no model is trained. Prompt engineering is a form of training: each prompt change chosen by looking at results fits the system to those examples. Divide the evaluation set into a development portion used while designing prompts and retrieval, and a held-out test portion used only for final evaluation. Without this separation, it is easy to write prompts that handle every development example and still fail on new questions.

Splits for small datasets

When data is very limited, for example a few hundred labelled examples, every split decision matters. Use cross-validation for model selection, keep the test set as large as practical, and report confidence intervals, which will be wide. Consider whether pretrained models, few-shot prompting or transfer learning can make better use of limited data than training from scratch. Above all, resist the temptation to evaluate on training data or to reuse the test set repeatedly, since with small data the resulting optimism can be very large.

A split design checklist

Before creating splits, answer a short checklist. What entity must the model generalize to: new records, new users, new sites, or future periods? Are there groups, near-duplicates or time dependencies? Which features would not be available at the moment of prediction? Which preprocessing steps learn from data and must be fitted on training data only? How large must the test set be for the required precision (Lesson 8)? Who is responsible for keeping the test set locked? Writing the answers in the data report makes the evaluation design explicit and reviewable.

Detecting leakage after the fact

Several checks can reveal leakage that slipped through. Unusually high performance compared with baselines, published work or human experts is the first warning. Feature importance analysis may show that the model relies heavily on a feature that should be weak, such as an identifier or a timestamp. Performance that drops sharply on a fresh, independently collected test set suggests the original evaluation was optimistic. Training a model to distinguish training from test examples, called adversarial validation, shows whether the splits differ in ways the main model might exploit.

When in doubt about a split, choose the more conservative design, the one that makes the test conditions most like real use. An honest but lower score is far more useful to the partner and to reviewers than an inflated one that collapses when the system meets real data, and it protects the team from having to retract results late in the term.