GCP-194 · Lesson 7 of 12 · 20 min

Review: Data management

▶ Video coming soon A video version of this lesson is on the way.

Section 4 is about 25 in 100 of the exam, roughly 15 questions on a 60-question form, almost as many as analysis. It was Mateo's weakest section in the diagnostic: 6 of 10. He had thought of governance as the part of the job other people did, Teresa Okonjo with access reviews and Daniel Ferreira with retention schedules. Ravi was blunt about it. "On this exam, the data practitioner is the person who sets the permissions and the lifecycle rules," he said. "If you leave a quarter of the marks to other people, you will not pass comfortably."

The section has four objectives in the copy of the guide we read: access control and governance; lifecycle management; high availability and disaster recovery; and encryption and compliance, which the guide numbers 4.2 a second time. This lesson takes them in that order.

Least privilege with IAM

Identity and Access Management controls who can do what to which resource. A principal, such as a user, a group or a service account, is granted a role, which is a bundle of permissions, on a resource, such as a project, a BigQuery dataset or a Cloud Storage bucket. Grants are inherited downward: a role on a project applies to every dataset and bucket in it. Least privilege means giving each principal only the permissions it needs, at the narrowest level that works.

Kinds of IAM role.
Kind of roleExamplesUse on the exam
Basic rolesOwner, Editor, ViewerAlmost always too broad; a red flag in a least-privilege question
Predefined BigQuery rolesBigQuery Data Viewer, Data Editor, Data Owner, Job User, User, AdminThe usual right answer, granted on a dataset or table where possible
Predefined Cloud Storage rolesStorage Object Viewer, Object Creator, Object User, Object Admin, Storage AdminChoose by whether the principal reads, writes or manages objects
Custom rolesA chosen list of permissionsWhen no predefined role is narrow enough

Two BigQuery details appear often. First, reading data and running queries are separate. BigQuery Data Viewer lets a principal read tables in a dataset; BigQuery Job User lets it run jobs, including queries, in a project, and is granted at the project level. An analyst who must query one dataset usually needs Data Viewer on that dataset and Job User on the project where queries run. Second, BigQuery Data Editor can change data and Data Owner can also manage access, so neither belongs to someone who only reads.

Worked problem

Granting the narrowest access

A finance analyst must query the curated dataset in project bl-data-prod and save results to a personal scratch dataset in the same project. She must not see the raw or staging datasets. Which roles, granted where, give her what she needs and nothing more?

Show solution
  1. To read curated tables: BigQuery Data Viewer on the curated dataset only, not on the project.
  2. To run queries: BigQuery Job User on the project, because jobs run at project level and the role grants no access to data.
  3. To save results: BigQuery Data Editor on her scratch dataset only.
  4. No basic roles, and nothing on raw or staging.

Answer: Data Viewer on curated, Job User on the project, Data Editor on her scratch dataset.

Birchline's access review in GCP-222 found the opposite pattern: 61 people could read the curated dataset, 23 of them through grants to named individuals rather than groups, and four pipeline service accounts held BigQuery Admin. The review replaced named grants with groups, narrowed the service accounts to the roles each pipeline used, and set expiry dates on temporary access with IAM conditions. Those three moves, groups rather than individuals, the narrowest role, and time limits for temporary access, are the habits the exam rewards.

Access to Cloud Storage

The guide asks candidates to compare methods of access to Cloud Storage. By default buckets and objects are private. Uniform bucket-level access means only IAM controls access to a bucket and everything in it; object-level access control lists are switched off. Google recommends it for most buckets because it is simpler to reason about. Fine-grained access keeps object ACLs as well, for the rare case where individual objects need different permissions. Public access prevention blocks any grant that would make data public. A signed URL gives anyone who has the link time-limited access to one object, without a Google account, which suits sharing a single file with an outside party.

Cloud Storage access methods matched to needs.
NeedMethod
Simple, consistent control over a whole bucketUniform bucket-level access with IAM roles
Make sure nothing in a bucket ever becomes publicPublic access prevention
Let a supplier download one file for 24 hours, without an accountA signed URL
Serve public product images on a websiteA public bucket, deliberately, with nothing else in it

Sharing data with BigQuery sharing

The guide asks when to share data with Analytics Hub, which Google now calls BigQuery sharing. A data owner publishes a listing of a dataset in an exchange. A subscriber, in another project or another organization, subscribes and gets a linked dataset: a read-only reference to the shared data, which they query in their own project and pay for with their own query costs. The data is not copied, so subscribers always see the current version, and the publisher can revoke access. It fits sharing data with partners, suppliers or other business units at scale. For sharing inside one team, a dataset grant or an authorized view is simpler.

Lifecycle management

Objective 4.2 asks candidates to choose Cloud Storage classes by access frequency and retention, to set deletion rules for BigQuery and Cloud Storage, and to evaluate archiving options.

Four columns for the Cloud Storage classes from left to right: Standard (frequent access, no minimum duration), Nearline (about once a month, 30-day minimum), Coldline (about once a quarter, 90-day minimum), Archive (less than once a year, 365-day minimum). An arrow underneath runs from higher storage price and no retrieval fee on the left to lower storage price and higher retrieval fees on the right. A note says Autoclass moves objects between classes automatically based on access.
The four Cloud Storage classes. Colder classes cost less to store and more to read, and have minimum storage durations.
Storage classes. Deleting an object before its minimum duration is billed as if it had been kept for the minimum.
ClassTypical accessMinimum storage durationExample
StandardFrequent, or kept only brieflyNoneThe landing zone for new till files
NearlineAbout once a month30 daysMonthly report extracts
ColdlineAbout once a quarter90 daysRaw files kept for audits
ArchiveLess than once a year365 daysTax records kept for many years

Object Lifecycle Management rules change an object's class or delete it when a condition is met, such as its age. Autoclass moves each object between classes based on how often it is actually read, which suits buckets where access is unpredictable. A retention policy stops objects from being deleted or replaced before a set period has passed, and locking the policy makes it permanent, which suits records that must be kept by law.

Worked problem

A lifecycle for raw files

Birchline keeps raw till files for 400 days. New files are read often for the first 30 days, then rarely, mostly for audits. The bucket's rule moves each file from Standard to Coldline at 30 days and deletes it at 400 days. How many days does each file spend in each class, and does the Coldline minimum cause an early-deletion charge?

Show solution
  1. Days in Standard: from day 0 to day 30, so 30 days.
  2. Days in Coldline: from day 30 to day 400, so 370 days.
  3. Coldline's minimum is 90 days, and 370 is more than 90, so no early-deletion charge applies.
  4. If the rule moved files to Archive at 30 days instead, they would spend 370 days there, which is above Archive's 365-day minimum, but audits would pay higher retrieval costs.

Answer: 30 days in Standard and 370 in Coldline, with no early-deletion charge.

BigQuery has its own deletion controls. A table expiration deletes a table at a set time; a default table expiration on a dataset applies to every new table in it, which suits scratch datasets such as Birchline's, where tables vanish after seven days. A partition expiration deletes partitions older than a set age, which suits keeping, for example, 25 months of web events in a date-partitioned table without deleting the table. Birchline's retention schedule from GCP-222 set sales and tax records at seven years, raw files at 400 days and web events at 25 months.

High availability and disaster recovery

Objective 4.3 asks candidates to compare Google-managed backup and recovery options for Cloud Storage and Cloud SQL, to decide when to use replication, and to distinguish location types for redundancy.

Recovery options in three services.
ServiceProtection against mistakesProtection against a regional outage
Cloud StorageSoft delete keeps deleted objects for a period (7 days by default, settable from 7 to 90); object versioning keeps older versionsDual-region or multi-region buckets keep copies in more than one region; turbo replication speeds that up
BigQueryTime travel lets you read or restore data as it was up to 7 days ago (settable from 2 to 7); table snapshots keep a copy for longerMulti-region datasets, or copies or replicas in another region
Cloud SQLAutomated backups and point-in-time recoveryHigh availability with a standby in another zone; cross-region read replicas for regional failures

Two pairs of ideas are easy to confuse. Backup protects against mistakes and corruption: you restore an earlier copy. High availability protects against failures: a standby takes over. Cloud SQL's high-availability configuration keeps a standby in another zone of the same region, written synchronously, and fails over in about a minute, but it does not protect against someone deleting a table, because the deletion is copied to the standby too. Replication across regions protects against the loss of a region. A question about recovering from an accidental deletion wants backups or time travel; a question about surviving a zone or region failure wants high availability or replication.

Encryption and keys

The last objective asks candidates to identify use cases for Google-managed, customer-managed and customer-supplied encryption keys, to understand the role of Cloud KMS, and to tell encryption in transit from encryption at rest. Google Cloud encrypts all data at rest and in transit by default. The options differ in who controls the keys.

A horizontal scale from more convenience on the left to more customer control on the right. Google-managed encryption keys (the default; nothing to set up). Customer-managed encryption keys (keys in Cloud KMS that the customer creates, rotates, disables and audits). Customer-supplied encryption keys (the customer provides the key with each request; Google does not store it; only some services such as Cloud Storage and Compute Engine). Below, a note: losing a customer-supplied key means losing access to the data.
Three key options, from most convenient to most customer control.
Encryption key options.
OptionWho controls the keyChoose it when
Google-managed (GMEK)Google creates, stores and rotates itNo rule requires anything else; the default for most data
Customer-managed (CMEK)The customer, in Cloud KMS: create, rotate, disable, destroy, auditPolicy or regulation requires control over keys, rotation schedules or the ability to revoke access
Customer-supplied (CSEK)The customer, outside Google; sent with each request and not storedThe key must never be held by Google; supported by a few services only

Cloud KMS is the service that holds customer-managed keys. A practitioner grants a service, such as BigQuery or Cloud Storage, permission to use a key, and every use is logged. Disabling the key makes the data unreadable until it is enabled again, which is both the point of CMEK and its main risk. Encryption at rest protects stored data; encryption in transit protects data moving across networks, and Google applies both by default.

Location as a compliance choice

Location appears in section 4 as well as in section 1. In section 1 it is about latency, cost and redundancy; in section 4 it is about rules. Some data must stay in a given country or region, for example because of a contract or a privacy law. A BigQuery dataset's location is fixed when the dataset is created and cannot be changed afterwards, so the decision has to be made first; moving data means copying it to a new dataset in the new location. A Cloud Storage bucket's location is also fixed at creation. Organization policies can restrict the locations in which resources may be created at all, which stops a well-meaning engineer from creating a bucket in the wrong place. On the exam, a scenario with a residency rule points to choosing a region or dual-region inside the required area and to a location restriction policy, not to a multi-region that may span a wider area. Birchline keeps its customer data in US locations for exactly this reason, as GCP-220 explained.

Common mistakes

  • MistakeGranting Editor because it is quick.

    InsteadGrant the narrowest predefined role on the narrowest resource.

  • MistakeGiving an analyst Data Viewer and expecting queries to run.

    InsteadRunning queries also needs Job User on the project.

  • MistakeMoving short-lived files to Archive to save money.

    InsteadArchive has a 365-day minimum; early deletion is billed as if kept the minimum.

  • MistakeRelying on Cloud SQL high availability to undo a deleted table.

    InsteadUse backups or point-in-time recovery; the standby copies the deletion.

  • MistakeChoosing customer-supplied keys for BigQuery.

    InsteadBigQuery supports customer-managed keys in Cloud KMS, not customer-supplied keys.

Summary and key terms

  • Section 4 is about 25 in 100 of the exam.
  • Grant predefined roles to groups at the narrowest level; basic roles are usually wrong.
  • Reading BigQuery data and running queries need separate roles.
  • Uniform bucket-level access, public access prevention and signed URLs control Cloud Storage access.
  • BigQuery sharing, formerly Analytics Hub, shares live datasets with other projects and organizations.
  • Storage classes trade storage price against retrieval cost and minimum durations; lifecycle rules, Autoclass, retention policies and BigQuery expirations manage data over time.
  • Backups undo mistakes, high availability survives failures and replication survives regional loss.
  • Google-managed keys are the default; customer-managed keys live in Cloud KMS; customer-supplied keys are held by the customer.
Least privilege
Granting only the permissions needed, at the narrowest level.
BigQuery Job User
A role that lets a principal run jobs, such as queries, in a project.
Uniform bucket-level access
A Cloud Storage setting where only IAM, not object ACLs, controls access.
Signed URL
A link that gives time-limited access to one Cloud Storage object.
Linked dataset
A read-only reference to a dataset shared through BigQuery sharing.
Minimum storage duration
The shortest period a storage class bills for, even if an object is deleted sooner.
Time travel
BigQuery's ability to read or restore table data as it was up to seven days earlier.
CMEK
Customer-managed encryption keys, created and controlled by the customer in Cloud KMS.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why is a grant of BigQuery Data Viewer on the project usually wrong for an analyst who needs one dataset?

    Show answer

    It gives read access to every dataset in the project.

  2. A supplier needs to download one price file this week without a Google account. What fits?

    Show answer

    A signed URL with a short expiry.

  3. Which protects against an accidental DELETE in Cloud SQL: high availability or point-in-time recovery?

    Show answer

    Point-in-time recovery; the high-availability standby copies the deletion.

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz