Security and privacy risks of generative AI

Teresa Okonjo, Birchline's information security manager, has a short list taped to her monitor: "What data goes in? Who can see what comes out? What can it do? Who can change it?" She has used those four questions for every system she has reviewed for ten years, and she uses them now for generative AI. In November 2026 she sits down with Jules and Farah to review the plan for the shopping assistant on the online shop, the customer-facing agent that GCP-102 placed behind Model Armor. "Generative AI does not replace security basics," she says. "It adds some new ways for things to go wrong, and it makes some old ones easier." This lesson covers those risks at the level leaders need, and the controls that address them. It is about defence: recognizing risks so they can be prevented.

What is different about generative AI security

Traditional applications follow code that developers wrote, so their behaviour is predictable. A generative AI application also follows instructions written in natural language, and it reads natural language from users, documents and websites. That blurs a line security teams rely on: the line between instructions and data. If a document the assistant reads contains text that looks like an instruction, the model may treat it as one. Generative AI applications also often hold more context than a typical app, such as grounding documents and conversation history, and agents can take actions through tools. Each of these widens what an attacker or an accident can affect.

Secure AI means designing, building and operating AI systems so that they resist attack and misuse, protect the data they use, and behave as intended across their whole life cycle. It is not a separate discipline from cybersecurity; it extends it.

The main risks, in plain terms

Jules is surprised that the last risk is one of the most common. Shadow AI usually comes from good intentions, people trying to work faster, and the best response is to provide approved tools that meet the need, together with clear rules, rather than only banning things. Birchline's acceptable use policy lists approved tools by data type, which Teresa calls the most effective security control the AI program has delivered so far.

Layered controls

No single control stops every generative AI risk, so Teresa plans defence in depth, several layers that each reduce the chance of harm.

• Identity and least privilege. The application and any agent run with service identities that can reach only what the task needs. The shopping assistant can read product data and order status for the signed-in customer; it has no permission to change or cancel orders. If it is tricked, there is little it can do.
• Data access at retrieval. Grounding respects each user's permissions, as lesson 5 required, so a prompt cannot pull in documents the user could not open.
• Input and output screening. Prompts and responses pass through screening that detects prompt injection and jailbreak attempts, harmful content categories, malicious links and sensitive data such as card numbers. On Google Cloud, Model Armor provides this kind of screening with templates that set filters and thresholds per application, and it can either log violations or block them.
• Sensitive data protection. Tools such as Google Cloud's Sensitive Data Protection find and de-identify personal information in documents before they are indexed, and in logs.
• Human confirmation for actions. Any action with consequences, such as a refund, requires the customer or a staff member to confirm explicitly.
• Logging and monitoring. Prompts, responses and tool calls are logged with personal data masked, and unusual patterns, such as many injection attempts from one account, raise alerts. Security Command Center gives a central view of security findings across Google Cloud projects.
• Secure foundations. The usual cloud controls still apply: encryption, network controls, patching, vulnerability management and audit logs.

Google's Secure AI Framework

Google published the Secure AI Framework (SAIF) in 2023 as a conceptual framework for securing AI systems, and the Generative AI Leader exam expects candidates to recognize its purpose. SAIF has six core elements, which Teresa summarizes for Jules in her own words.

The last element connects security to the rest of this course. The shopping assistant and the copy team's drafting tool use similar technology, but the shopping assistant faces the public and touches customer accounts, so it gets stronger controls, red-team testing before launch and a faster incident path.

Testing before launch: red teaming

Red teaming means deliberately trying to make a system fail in the ways an attacker or a careless user might, before it reaches the public. For the shopping assistant, Teresa's team spends two weeks trying prompt injection through chat and through product reviews the assistant might read, trying to extract other customers' data, trying to get it to promise discounts, and trying off-topic and harmful requests. They record each finding, its severity and the fix. The assistant launches only after every high-severity finding is fixed and retested. Red teaming is repeated when the model, prompt or tools change.

Privacy risks and principles

Privacy overlaps with security but is a distinct duty: using personal information only in ways people would reasonably expect, with a lawful basis, and no more than needed. Generative AI raises privacy questions at every stage: personal data in grounding documents, personal data typed into prompts, outputs that reveal personal details, and logs that keep conversations.

In Canada, private-sector organizations are governed by the Personal Information Protection and Electronic Documents Act (PIPEDA) in most provinces, with substantially similar provincial laws in Quebec, Alberta and British Columbia. Quebec's Law 25 adds specific duties, such as privacy impact assessments for certain projects, an assessment before personal information is communicated outside Quebec, and telling people when a decision about them is made exclusively by automated processing. Birchline treats Quebec customers' data with those duties in mind, and its privacy officer reviews every customer-facing AI use case.

Two privacy techniques appear in the exam guide and in everyday practice. Anonymization removes or transforms personal information so that individuals can no longer be identified, by anyone, by reasonable means. Pseudonymization replaces identifiers with codes, such as replacing a customer's name with a token, so that the data cannot be linked to a person without separate information that is kept secure. Pseudonymized data is still personal information in many legal frameworks, because re-identification is possible; truly anonymized data generally is not, but true anonymization is harder than it looks, especially with rich text.

import re, hashlib
SECRET = "rotate-this-salt"   # illustrative; real systems keep it in a secrets manager
def pseudonym(value):
    return "CUST-" + hashlib.sha256((SECRET + value).encode()).hexdigest()[:8]
email = re.compile(r"[\w.]+@[\w.]+")
phone = re.compile(r"\b\d{3}[-. ]\d{3}[-. ]\d{4}\b")
note = "Ana Silva (ana.silva@example.com, 416-555-0142) wants to return boots from order 88231."
masked = email.sub(lambda m: pseudonym(m.group()), note)
masked = phone.sub("[PHONE]", masked)
masked = masked.replace("Ana Silva", "[NAME]")
print(masked)

[NAME] (CUST-b70d5ea1, [PHONE]) wants to return boots from order 88231.

The toy example replaces the email with a consistent pseudonym, so the same customer always gets the same code, and masks the phone number and name. Real de-identification uses tested tools, because names and identifiers appear in many forms. The business lesson is the order of operations: remove personal information that the task does not need before it reaches prompts, indexes or logs.

Questions every leader should ask

Teresa gives Jules a short list to ask of any generative AI proposal, internal or from a vendor.

• What data goes in, and is any of it personal or confidential? Is all of it needed?
• Where is data processed and stored, and is customer data used to train the provider's models? (Enterprise cloud AI services commonly commit not to, but leaders should confirm it in the terms.)
• Who can see outputs and logs, and for how long are they kept?
• What actions can the system take, and what is the worst a misled system could do?
• How are prompts and responses screened, and what is logged when something is blocked?
• Has it been red-teamed, and how often will that repeat?
• Who is called when something goes wrong, and how fast?

Incident response for AI systems

Even with good controls, something will eventually go wrong, so Birchline's security incident plan now covers AI systems explicitly. If the shopping assistant is found to have revealed information it should not, or to have been misled into harmful output, the plan says who is called, how the assistant can be switched to a safe fallback message within minutes, how logs are preserved for investigation, and when the privacy officer must assess whether a privacy breach has occurred and whether it must be reported. In Canada, PIPEDA requires organizations to report breaches of security safeguards involving personal information that create a real risk of significant harm, and to notify affected people. US state breach-notification laws impose similar duties with their own definitions and deadlines. Rehearsing the plan once, in a short tabletop exercise, showed the team that nobody knew who could switch the assistant off at night, which was fixed the same week.