Data: sourcing, permissions, quality and documentation

Data determines what a model can learn, and in most capstone projects it consumes more time than modeling. Finding suitable data, confirming that you are allowed to use it, cleaning and labeling it, and documenting what it contains are skilled tasks with legal, ethical and technical dimensions. Problems that are not caught early, such as missing permissions, biased sampling, label noise or duplicated records, can invalidate every result that follows. This lesson covers sources of data, licenses and consent, privacy law and research ethics, data quality auditing, labeling and annotation, and documenting datasets with datasheets.

Sources of data

Capstone data typically comes from one or more of the following: data provided by a project partner, such as records, documents or images from its operations; public datasets from repositories such as government open data portals, research archives and benchmark collections; data the team collects, through surveys, sensors, photography or web forms; data obtained through public interfaces, such as application programming interfaces (APIs); and synthetic data generated by simulation or models. Each source has advantages and pitfalls. Partner data is realistic but may be messy, sensitive and slow to obtain. Public benchmarks are convenient but may not represent the real use case and are often already "solved". Collected data fits the need but is expensive. Synthetic data is cheap but may not reflect real-world variation, and models trained on it may fail on real inputs.

Licenses and terms of use

Data is rarely free of conditions. Public datasets carry licenses, such as Creative Commons licenses, open government licenses or custom research licenses, that may restrict commercial use, require attribution, prohibit redistribution or forbid particular uses. Websites have terms of service that may prohibit automated collection, and copyright protects much text, imagery and audio. Partner data comes with contractual restrictions on use, storage and sharing. Before using any dataset, record its source, license or agreement, permitted uses and obligations, and check that the capstone's intended uses, including public demonstration and publication, are allowed. When in doubt, ask the program's legal or research office.

Privacy and personal information

Data about identifiable people is regulated, and capstone teams must handle it lawfully and carefully.

In the United States, there is no single comprehensive federal privacy law. Sector laws apply, such as HIPAA for health information held by covered entities, FERPA for student education records and COPPA for children's online data, and a growing number of states, led by California's CCPA as amended by the CPRA, have comprehensive consumer privacy laws. Research involving human subjects at institutions receiving federal funding follows the Common Rule, which requires review by an institutional review board (IRB) for many projects.

Practical steps for any project using personal data include collecting only what is necessary; removing direct identifiers such as names, addresses and identification numbers, while recognizing that de-identified data can sometimes be re-identified by combining fields; storing data securely with access limited to the team; avoiding the upload of personal data to external services, including commercial AI APIs, unless the agreement and law permit it; and planning for deletion at the end of the project. Programs often require teams to complete an ethics or data management form before accessing sensitive data.

Auditing data quality

Every dataset should be audited before modeling. A basic audit checks: the number of records and whether it matches expectations; missing values in each field and whether missingness is random or patterned; duplicates, which inflate apparent performance if they cross train and test splits; label distribution and class balance; distributions of key features, looking for impossible values, outliers and unit errors; representation of important subgroups, such as regions, devices or demographic groups; and label quality, estimated by having a sample labeled twice and measuring agreement.

import numpy as np
# A quick data audit: missing values, duplicates, label balance and label noise
rng = np.random.default_rng(1)
n = 1200
age = rng.normal(45, 15, n).round(); age[rng.random(n) < 0.06] = np.nan
income = rng.lognormal(10.8, 0.5, n); income[rng.random(n) < 0.15] = np.nan
region = rng.choice(["north", "south", "east", "west"], n, p=[0.55, 0.25, 0.15, 0.05])
label = rng.random(n) < 0.12
dup_idx = rng.choice(n, 40, replace=False)
customer = rng.integers(10**8, 10**9, n)        # customer identifier
rows = np.column_stack([customer, np.nan_to_num(age, nan=-1), np.nan_to_num(income, nan=-1)])
rows[dup_idx[20:]] = rows[dup_idx[:20]]          # inject 20 exact duplicates
print(f"rows: {n}")
for name, col in [("age", age), ("income", income)]:
    print(f"missing {name}: {np.mean(np.isnan(col)):.1%}")
_, counts = np.unique(rows, axis=0, return_counts=True)
print(f"duplicate rows: {int(np.sum(counts[counts > 1] - 1))}")
print(f"positive labels: {label.mean():.1%}")
for r in ["north", "south", "east", "west"]:
    m = region == r
    print(f"  region {r:5s}: {m.sum():4d} rows, positives {label[m].mean():.1%}")
# Label noise estimate from a double-labeled sample
a = label[:200].copy(); b = a.copy(); flip = rng.random(200) < 0.07; b[flip] = ~b[flip]
print(f"annotator agreement on 200 double-labeled rows: {np.mean(a == b):.1%}")

rows: 1200
missing age: 6.8%
missing income: 15.5%
duplicate rows: 20
positive labels: 11.0%
  region north:  628 rows, positives 10.0%
  region south:  321 rows, positives 11.5%
  region east :  191 rows, positives 14.7%
  region west :   60 rows, positives 6.7%
annotator agreement on 200 double-labeled rows: 93.5%

The audit reveals issues that would shape every later step. Income is missing for about 15 % of rows, which must be handled deliberately rather than by silently dropping rows. Twenty duplicate rows must be removed or kept within a single split. The western region has only 60 rows, so any results for it will be very uncertain. Annotators agree on about 92 % of double-labeled rows, implying that around 8 % of labels may be wrong or ambiguous; a model cannot be expected to exceed this level of agreement by much, and the labeling guidelines may need clarification.

Labeling and annotation

Many projects need labels that do not yet exist. Good annotation starts with written guidelines that define each label, give examples and counter-examples, and explain how to handle ambiguous cases. Annotators should be trained on the guidelines, then label a shared pilot set so that disagreements can be discussed and guidelines refined. Inter-annotator agreement, measured by simple agreement or chance-corrected statistics such as Cohen's kappa, indicates whether the task is well defined. Annotation tools, such as open-source labeling platforms, speed the work. Where labels are expensive, active learning, which selects the most informative examples for labeling, and weak supervision, which combines noisy heuristic labels, can reduce effort. Annotators should be treated fairly: if the program uses paid annotators or crowd workers, pay and working conditions matter, and annotators of distressing content need support.

Documenting datasets

Datasets should be documented so that others, and your future selves, know what they contain and how they may be used. Gebru and colleagues proposed datasheets for datasets in 2018, a set of questions covering motivation (why was the dataset created?), composition (what does it contain, and what is missing?), collection process (how, when and by whom was it collected, with what consent?), preprocessing and labeling, uses (what is it suitable and unsuitable for?), distribution and maintenance. Similar formats include data cards and data statements for language data. A short datasheet is a required part of the capstone data report, and writing it often reveals gaps, such as unknown collection dates or unclear consent, that need to be resolved.

Handling missing data and outliers

Missing values and outliers must be handled deliberately. Options for missing values include removing rows (acceptable only when few are missing and missingness is unrelated to the outcome), imputing values with simple statistics or models, adding indicator features that flag missingness, which lets the model learn whether missingness itself is informative, and using models that handle missing values natively. Outliers may be errors to correct or remove, or genuine rare cases that matter most, such as large fraudulent transactions; domain experts can tell which. Every such decision should be applied identically to training and evaluation data, fitted only on the training split (Lesson 4), and recorded.

Synthetic data and data generated by models

Synthetic data, whether from simulation or from generative models, can supplement scarce real data, protect privacy and create examples of rare cases. It has risks: generated examples may lack the diversity and messiness of real inputs, may encode the generating model's biases, and can make evaluation misleading if test sets are synthetic too. A sound approach uses synthetic data mainly for training augmentation, always evaluates on real data, measures whether adding synthetic data actually improves real-data performance, and documents how synthetic examples were created. Labels produced by large language models can speed annotation, but their accuracy must be checked against human labels on a sample, as with any annotator.

Data management during the project

Data must be managed throughout the term, not only at the start. Keep raw data unchanged in a protected location and derive cleaned versions with scripts, so that every processing step can be repeated. Version datasets, for example with dated folders or data versioning tools, and record which version each experiment used. Restrict access to sensitive data to team members who need it, avoid copies on personal laptops and in email, and use the storage provided by the program or partner. At the end of the project, return or delete data as agreed, and record that this was done.

Representativeness

A dataset can be large and clean yet unrepresentative of the people or situations where the system will be used. Ask how the data was collected, who or what is missing, and whether conditions at deployment differ, such as different cameras, languages, regions or time periods. Compare the dataset's composition with the target population where possible, and plan slice evaluations (Lesson 6) for groups that are under-represented. Unrepresentative data is one of the main sources of unfair performance and of models that fail when deployed.

Data agreements in practice

When a partner provides data, a data sharing agreement usually specifies what data is shared, for what purpose, who may access it, where it must be stored, whether it may leave the country or be processed by third-party services, how long it may be kept, how it must be destroyed, and what happens in case of a breach. Read the agreement carefully before planning the technical approach, because a restriction such as "data may not be sent to external services" rules out commercial AI APIs and may require self-hosted models.

Finally, keep a short data log alongside the datasheet, recording each change to the data, such as relabeling rounds, removed duplicates, new collection batches and corrected errors, with dates and reasons. When results change unexpectedly, the data log is often where the explanation is found, and it allows reviewers to see exactly which version of the data supports each reported result.