Dataplex and data governance

On Monday, November 2, the session moves from keeping pipelines running to keeping data trustworthy. Teresa Okonjo, Birchline's information security manager, joins Aisha and Mateo for the first half hour. Birchline is about to give store managers direct access to more sales and staffing data, and Teresa wants the data team to be able to answer three questions about any table before that happens: who owns it, how sensitive it is, and how good it is. Answering those questions consistently, for every table, is what data governance means in practice.

What data governance is

Data governance is the set of roles, rules and tools an organization uses to make sure its data is known, trusted, protected and used appropriately. It is not a product and not a one-off project. Aisha describes it as the difference between a library and a pile of books: the books are the same, but in a library each one has a place, a catalog entry, a condition and a set of borrowing rules.

Governance has three parts that must work together. People: every important dataset has a data owner, a business leader accountable for it, and a data steward, someone who looks after it day to day, answers questions and approves access. Rules: a classification scheme, quality expectations, retention periods and access policies, written down and agreed. Tools: a catalog that records what exists and who owns it, quality checks that measure whether data meets its expectations, and access controls that enforce the policies. Tools without people produce a catalog nobody maintains, and rules without tools are guidelines that nobody can check; people without tools produce a spreadsheet that is out of date the day after it is written.

Classification

Teresa's first rule is a classification scheme: a short list of sensitivity levels, each with handling requirements. Birchline uses four. Public data, such as store addresses and opening hours, can be shared with anyone. Internal data, such as sales totals by store, is for staff only. Confidential data, such as supplier costs and staff schedules, is limited to the teams that need it. Restricted data, such as customer contact details, payment references and employee pay, has the tightest controls, is masked for most users and is logged on every access.

Each table is classified by its highest level column: curated.sales is internal, but a returns table with customer email addresses is restricted even though most of its columns are not. That is why column-level controls, covered in lessons 8 and 10, matter: they let most users see a restricted table's harmless columns without seeing the sensitive ones.

Dataplex and the Knowledge Catalog

Dataplex is Google Cloud's family of data governance services. Its parts have been renamed several times, so it helps to know the current names and the old ones. The catalog at its center, which records every BigQuery dataset and table, every Cloud Storage bucket registered with it, and much more, was called Dataplex Universal Catalog until April 10, 2026, when Google renamed it Knowledge Catalog. The older standalone Data Catalog service was deprecated in 2025 and shut down on June 1, 2026, with its contents moved into the newer catalog. APIs, gcloud dataplex commands and IAM roles still use the dataplex name, so the exam and the documentation will show both.

Alongside the catalog, Dataplex provides data profiling scans, which describe what a table actually contains, data quality scans, which test it against rules, data lineage, which records how data moves between tables, and discovery, which can find structured files in Cloud Storage and register them as tables that BigQuery can query. Earlier versions organized data into lakes and zones, logical groupings of buckets and datasets with shared settings; Birchline does not use them, and Google's current guidance centers on the catalog, scans and policies instead.

Profiling: what is really in a table

A data profile scan reads a table, or a sample of it, and reports for each column the share of nulls, the number of distinct values, the minimum, maximum and average for numbers, the most common values and the typical length of strings. It answers questions people often guess at: is customer_email really filled in for every return, how many distinct store IDs appear, what range do discount amounts fall in?

Profiles are the starting point for quality rules. When Mateo profiles the returns table, he finds that reason_code is null in about 6 of every 100 rows and that 12 distinct codes appear, although the returns policy lists only 10. Two of the codes, it turns out, belong to an old version of the returns portal. Without the profile, nobody would have written a rule for them.

Data quality scans

A data quality scan checks a table against a set of rules and records the results. Rules come in a few types: a column must not be null, must be unique, must fall in a range, must be one of a set of values, must match a pattern, or must satisfy a SQL expression; table-level rules can check row counts or any SQL condition. Each rule has a dimension, such as completeness, validity, uniqueness or freshness, so results can be summarized by kind, and a passing threshold: the share of rows that must pass for the rule to pass. A scan can run on a schedule or on demand, scan the whole table or only new data, publish its results to the catalog entry, and write detailed results to a BigQuery table for dashboards and alerts.

GCP-220 set up the first quality scan on staging.pos_sales. Today Aisha adds the governance view: every curated table that a dashboard depends on has a scan, every scan has an owner who is told when it fails, and the scan results appear on the table's catalog entry so that an analyst opening the table can see at a glance whether yesterday's data passed. Mateo tries the rule logic on a sample of returns rows to see how thresholds behave.

# Evaluate three data quality rules against sample rows, with passing thresholds
rows = [
    {"return_id": 1, "store_id": 12, "reason": "SIZE",  "amount": 89.0},
    {"return_id": 2, "store_id": 15, "reason": None,    "amount": 45.5},
    {"return_id": 3, "store_id": 15, "reason": "FAULT", "amount": -10.0},
    {"return_id": 4, "store_id": 31, "reason": "SIZE",  "amount": 120.0},
    {"return_id": 5, "store_id": 0,  "reason": "LATE",  "amount": 64.0},
    {"return_id": 6, "store_id": 22, "reason": "OLD7",  "amount": 30.0},
    {"return_id": 7, "store_id": 22, "reason": "SIZE",  "amount": 15.0},
    {"return_id": 8, "store_id": 40, "reason": "FAULT", "amount": 52.0},
]
valid_reasons = {"SIZE", "FAULT", "LATE", "CHANGED_MIND"}
rules = [
    ("reason is not null", lambda r: r["reason"] is not None, 0.95),
    ("reason in policy list", lambda r: r["reason"] is None or r["reason"] in valid_reasons, 0.90),
    ("amount greater than zero", lambda r: r["amount"] > 0, 1.00),
]
for name, test, threshold in rules:
    passed = sum(1 for r in rows if test(r))
    share = passed / len(rows)
    verdict = "PASS" if share >= threshold else "FAIL"
    print(f"{name:26s} {passed}/{len(rows)} rows = {share:.3f}, needs {threshold:.2f}: {verdict}")

reason is not null         7/8 rows = 0.875, needs 0.95: FAIL
reason in policy list      7/8 rows = 0.875, needs 0.90: FAIL
amount greater than zero   7/8 rows = 0.875, needs 1.00: FAIL

All three rules fail on the sample, each because of one row out of eight, which is 0.875 of the rows passing. On the real table the thresholds are set from the profile: completeness at 0.95 because about 6 rows in 100 have no reason, which the returns team has agreed to reduce, and validity at 0.90 until the two old codes are mapped to current ones. The negative amount rule has a threshold of 1.00 because a negative refund is always an error. Thresholds are agreements, not physics: the steward sets them with the owner, and tightens them as the data improves.

Roles in practice

Aisha closes by showing how the roles map onto Google Cloud. Each data domain, such as sales, products, people or customers, is represented in the catalog by an entry group or by labels, with the owner and steward recorded as metadata on each table. Stewards are granted roles that let them edit catalog metadata and manage quality scans but not change the data. Owners approve access requests, which the platform team then grants through groups. A quarterly review lists every restricted table, its readers and its last quality result, and the owner signs it off. Teresa's three questions, who owns it, how sensitive is it and how good is it, can now be answered from the catalog entry of any curated table.

Discovery of files in Cloud Storage

Not all of Birchline's data lives in BigQuery tables. The landing bucket holds thousands of CSV and JSON files, and the warehouse team keeps Parquet exports in another bucket. Dataplex discovery can scan a bucket, recognize files with the same structure, infer their schema and register them in the catalog as tables, which BigQuery can then query as BigLake tables without loading them. The scan runs on a schedule and picks up new folders as they appear.

Discovery makes files findable and governable, which is its real value. Once the warehouse exports appear in the catalog, they have an entry with an owner, a classification and, if needed, a quality scan, just like a native table. Aisha's rule is that any bucket holding data that people query must either be registered this way or be loaded into BigQuery; data that only a single pipeline reads, such as the raw landing files, can stay unregistered, because its contents are governed through the tables it feeds.