The value of data and the data lifecycle

Ravi Menon, Birchline's data and analytics lead, has a joke he tells new staff: "We have more data than we have ever had, and we know less than we would like." Every till in forty stores records every sale. The online shop logs every search, every basket and every abandoned checkout. The warehouses scan every carton in and out. Yet when the buying team asks a simple question on a Monday morning, such as which tents sold best in the Prairies last weekend compared with the coast, the answer usually arrives on Wednesday, after the old data warehouse has finished loading and someone has built a spreadsheet. Owen joined Ravi for a week to understand why. This lesson follows what he learned about what makes data valuable and how it moves from the moment it is created to the moment someone acts on it.

Data transformation is the second section of the Cloud Digital Leader exam guide, and it begins with ideas rather than products. Before choosing BigQuery or Cloud Storage, a digital leader has to understand what kinds of data an organization has, where it comes from, how it should be stored and who is responsible for it. Lesson 4 then matches those needs to Google Cloud's products.

Why data matters to the business

Data creates value when it changes a decision or an action. The exam guide groups the benefits into a few themes. Real-time insight lets a business react while something is still happening, such as moving stock between stores during a heat wave when sales of fans and coolers jump. Trends emerge when data is collected over time, such as the slow shift of Birchline's customers toward lighter camping gear. Better decisions come from replacing opinion with evidence: the buying team can see which products are profitable after returns rather than which merely sell well. And data increasingly fuels AI, because machine learning models learn from examples, and the quality of those examples sets the limit on how good a model can be.

Ravi's point to Owen was that none of this happens because data exists. It happens because data reaches the right people quickly, in a form they trust. Birchline's problem is not a shortage of data. It is that sales, web and warehouse data sit in separate systems, arrive at different times and use different product codes, so combining them takes days of manual work. Organizations call this a data silo problem, and breaking silos down is one of the main reasons companies move their data to the cloud.

Databases, data warehouses and data lakes

Three kinds of data store appear again and again. A database stores the current data an application needs to do its job, such as the online shop's product catalog, customer accounts and orders. It is optimized for many small, fast reads and writes: look up one product, add one order. A data warehouse stores large amounts of cleaned, structured historical data from many sources, organized for analysis: total sales by region, by week, for the last five years. It is optimized for big queries that scan millions of rows. A data lake stores large amounts of raw data in its original format, structured or not, such as web logs, images, sensor readings and exported files, so that it can be processed later for purposes that may not be known yet.

The boundaries have blurred in recent years. Modern cloud data warehouses can query files sitting in a data lake directly, and the term lakehouse describes designs that combine the cheap, flexible storage of a lake with the structure and performance of a warehouse. For the exam, the key is to recognize which job each kind of store does best. A question that describes an application needing fast lookups of individual records points to a database; one that describes analysts running large queries over years of history points to a warehouse; one that describes keeping raw files of many kinds for future use points to a lake.

Kinds of data

Data can be classified by its structure. Structured data fits neatly into rows and columns with a fixed schema, like a sales table with a date, store, product code, quantity and price. Semi-structured data has some organization but no rigid table shape, such as JSON order records from the shop's API, where one order may have two items and another twenty, or an email, which has fixed headers and free text. Unstructured data has no predefined model at all: photos, videos, audio, scanned documents and free-text product reviews. Most of the world's data is unstructured, and until recently it was the hardest to use. AI has changed that, because models can now describe images, transcribe calls and summarize reviews at scale.

Data can also be classified by where it comes from. First-party data is collected directly by the organization from its own customers and operations: Birchline's sales, loyalty accounts and website behavior. Second-party data is another organization's first-party data shared through a direct partnership, such as sell-through figures from a tent manufacturer that sells through Birchline. Third-party data is collected by an outside party that has no direct relationship with the people it describes and is sold or licensed, such as regional weather histories or demographic data. First-party data is usually the most valuable and the most trusted, but it also carries the strongest privacy obligations, because customers gave it to Birchline directly.

The data supply chain

Google's exam guide describes data moving through a data supply chain, a useful way to see where value is added and where it can be lost. Data begins at genesis, the moment it is created: a sale at a till, a click on a product page, a scan at the warehouse dock. It is then collected, gathered from those sources into one place, either in batches or as a continuous stream. It is processed: cleaned, checked, converted to common formats and joined with other data, such as matching store product codes to web product codes. It is stored in the right kind of store for how it will be used. It is analyzed, through queries, dashboards and models, to produce insight. And finally it is activated: the insight is put to work, perhaps as an automatic stock transfer, a personalized email or a change to next season's buying plan.

Owen noticed that Birchline's chain breaks in two places. Collection happens once a night, in batch, so nothing is current before the morning. And processing is partly manual, because product codes differ between systems and someone fixes them in spreadsheets. Moving the warehouse to the cloud will not fix either problem by itself. Farah's plan adds streaming collection for sales and web events and an automated processing step that maps product codes once, so that every later report uses the same definitions.

Activation is the step most organizations neglect. A dashboard that shows stock running low in the Prairie stores creates no value until someone moves stock or the system does it automatically. Ravi's goal for the first year is modest: one automated action, a daily stock rebalancing suggestion sent to the regional managers, built on data they trust.

Data quality

Data that is wrong, late or incomplete leads to wrong decisions, and in AI systems it leads to models that confidently repeat the errors. Google's guide lists six dimensions of data quality. Completeness asks whether all the required data is present. Uniqueness asks whether each real thing is recorded once rather than duplicated. Timeliness asks whether the data is available when it is needed. Validity asks whether values follow the expected rules and formats. Accuracy asks whether the values match reality. Consistency asks whether the same fact is recorded the same way across systems.

# Simple quality checks on a few invented online orders.
orders = [
    {"id": "A101", "customer": "C7", "postal": "M5V 2T6", "price": 189.0},
    {"id": "A102", "customer": "C9", "postal": "",        "price": 64.5},
    {"id": "A103", "customer": "C7", "postal": "K1A 0B1", "price": 0.0},
    {"id": "A101", "customer": "C7", "postal": "M5V 2T6", "price": 189.0},
]
ids = [o["id"] for o in orders]
missing_postal = [o["id"] for o in orders if not o["postal"]]
duplicates = sorted({i for i in ids if ids.count(i) > 1})
bad_price = [o["id"] for o in orders if o["price"] <= 0]
print("Completeness - missing postal code:", missing_postal)
print("Uniqueness   - duplicated order IDs:", duplicates)
print("Validity     - impossible prices:   ", bad_price)
print("Orders passing all three checks:", len(orders) - len(set(missing_postal + duplicates + bad_price)), "of", len(orders))

Completeness - missing postal code: ['A102']
Uniqueness   - duplicated order IDs: ['A101']
Validity     - impossible prices:    ['A103']
Orders passing all three checks: 1 of 4

Checks like these run automatically in a modern pipeline, so that problems are caught at the processing step rather than discovered by an annoyed executive looking at a dashboard. Measuring quality also tells a team where to spend effort. Birchline found that most of its consistency problems came from one source, the product codes, which made the case for fixing that one thing first.

Data governance and openness

Data governance is the set of policies, roles and processes that decide how data is collected, stored, protected, used and eventually deleted. It answers questions such as who owns each dataset, who may see customer information, how long sales records are kept, how personal data is handled under federal and state privacy laws, and how quality problems are reported and fixed. Good governance is not a brake on using data; it is what makes people confident enough to use it. Teresa and Ravi now co-chair a small data governance group at Birchline that names an owner for every important dataset.

Governance also covers the risk of lock-in. Data stored in proprietary formats that only one product can read is hard to move, combine or reuse. The exam guide stresses openness and interoperability: open file formats, standard query languages such as SQL, and tools that can read data wherever it sits. Open formats help an organization avoid both data silos inside the company and dependence on a single vendor outside it.

Treating important data as a product

Ravi borrowed an idea that many data teams now use: treat each important dataset as a data product, with an owner, a clear description, documented quality checks and promised update times, just like a product sold to customers. The "customers" are the people inside Birchline who rely on it. The daily sales dataset, for example, now has a named owner in finance, a written definition of what counts as a sale (are returns subtracted, and when?), a target of being complete by six in the morning, and a page that shows whether today's checks passed.

The approach changes conversations. Instead of arguing about whose spreadsheet is right, teams point to the agreed definition. Instead of discovering problems in a board meeting, the owner sees a failed check at dawn and fixes it or warns users. And when the AI team wants training data, it starts from datasets that are already documented and trusted. For a digital leader, the lesson is that data value depends as much on clear ownership and agreed meanings as on technology.