Data tiers and retention

Nadia's question from the first day had two halves: what will we need to see, and how long will we need to see it for. By Monday, November 9, the first half was answered by the connectors. The second half filled that morning's session, and it turned out to be mostly about money. Keeping every table searchable at full speed for two years would have cost several times the rest of the Sentinel budget. Keeping nothing beyond the free period would have left investigations blind. The answer lay in using different tiers for different data.

Three places data can live

Microsoft now describes security data in the Defender portal in terms of tiers. The analytics tier is the Sentinel workspace as analysts have always used it: data there can be queried quickly, drives analytics rules, workbooks and hunting, and is billed for ingestion. The data lake tier is the Microsoft Sentinel data lake, a lower-cost store designed for keeping large volumes for a long time and querying them less often, with KQL or with Jupyter notebooks. The XDR tier is the Defender XDR data that advanced hunting can query in the Defender portal for about 30 days without it being ingested into Sentinel at all.

Two facts about the lake shaped the design. First, when the lake is enabled, data in the analytics tier is mirrored into the lake, so there is a single copy of each record: a table can keep 90 days in the analytics tier and two years in total, with the older data living only in the lake. Second, a table can be set to the lake tier only. Its data is then not in the analytics tier at all, which is much cheaper for high-volume data, but analytics rules cannot use it. You can bring selected lake data back into the analytics tier with a KQL job that runs once or on a schedule.

Retention settings

Retention is set per table. Analytics retention is how long data stays in the analytics tier; total retention is how long it is kept at all, including time in the lake. Data in the analytics tier is retained free for 90 days in a Sentinel workspace, and retention beyond that is charged at Log Analytics retention prices. Once the lake is onboarded, the extra time beyond analytics retention is held in the lake and billed by the lake's storage meter. Retention for many tables is managed in the Defender portal under data management for tables, where each table shows its tier, its analytics retention and its total retention.

Nadia set requirements by asking what investigations look like at Harrow & Finch. Most incidents are triaged within days, so 90 days of fast access covered almost all daily work. But two kinds of question reach further back. A phished account found in March might have been compromised in December, and the investigation needs the sign-ins from then. And the company's insurer and auditors expect to be able to ask what happened during an incident up to about two years later. So the rule she wrote was: 90 days in the analytics tier for anything rules use, two years in total for identity and audit data, and one year for network data.

The last row is the lake-only case. The cloud team had asked for Azure Firewall's verbose flow logs, about 7 GB a day, for occasional network investigations. No analytics rule needed them, and in the analytics tier they would have nearly doubled the bill. In the lake only, they would cost a fraction as much and could still be searched, or promoted into the analytics tier for an investigation, when needed. Incidents and alerts, by contrast, stay fully in the analytics tier for two years, because they are small and the SOC's own reports in lesson 12 need them.

How the lake is billed

The lake has its own meters, and they behave differently from analytics ingestion. Tables set to the lake only are billed for lake ingestion and for data processing, such as transformations, per gigabyte. Data kept in the lake, whether it was mirrored from the analytics tier or sent there directly, is billed for storage per gigabyte per month after the analytics retention period, and Microsoft bills storage with a uniform compression ratio of 6 to 1, so 600 GB of raw data is billed as 100 GB. Queries and KQL jobs against the lake are billed per gigabyte of uncompressed data scanned, and notebook sessions per compute hour.

The query meter changes how analysts should work. In the analytics tier a careless query costs only time; in the lake, scanning two years of sign-ins to find one user costs money every time. Sunil wrote a short guide for the team: narrow the time range first, filter on the most selective column, and use a KQL job to promote a small, filtered result into the analytics tier if an investigation will query it repeatedly.

# Illustrative prices only; real prices vary by region and change over time.
analytics_payg = 5.20          # dollars per GB ingested, pay-as-you-go
tier_100_per_day = 296.0       # dollars per day for a 100 GB/day commitment tier
lake_storage = 0.026           # dollars per GB-month, billed after 6:1 compression

billable_gb_day = 16.1
print(f"Pay-as-you-go per day: {billable_gb_day * analytics_payg:6.2f} dollars")
print(f"100 GB/day tier per day: {tier_100_per_day:6.2f} dollars")
print(f"Break-even volume: {tier_100_per_day / analytics_payg:.1f} GB per day")

# Lake storage for data kept 2 years in total, 90 days of it in the analytics tier
monthly_raw = billable_gb_day * 30
months_in_lake = 24 - 3
billed_gb = monthly_raw * months_in_lake / 6
print(f"Raw data per month: {monthly_raw:.0f} GB")
print(f"Lake storage billed at steady state: {billed_gb:,.0f} GB")
print(f"Lake storage per month: {billed_gb * lake_storage:.2f} dollars")

Pay-as-you-go per day:  83.72 dollars
100 GB/day tier per day: 296.00 dollars
Break-even volume: 56.9 GB per day
Raw data per month: 483 GB
Lake storage billed at steady state: 1,691 GB
Lake storage per month: 43.95 dollars

Pay-as-you-go or a commitment tier

Analytics ingestion is billed either pay-as-you-go, per gigabyte, or through a commitment tier, where you commit to a daily volume at a lower effective price per gigabyte. Commitment tiers start at 100 GB a day. Ingestion above the commitment is billed at the tier's effective rate, you can move to a higher tier at any time, and you can move to a lower tier only every 31 days. A tier pays off only when daily volume reliably exceeds its break-even point.

With about 16 GB a day of billable analytics data, Harrow & Finch was far below the smallest tier, so pay-as-you-go was the obvious choice. Sunil wrote down the break-even point anyway. If the company ever ingested DeviceFileEvents and the verbose firewall logs into the analytics tier, volume could pass 50 GB a day, and the comparison would need to be done again with real prices from the calculator.

Reducing volume without losing what matters

Tiers are one way to control cost; reducing what is ingested is the other. The team now had four techniques, and Ama wrote them down in order of preference, from least to most risky for investigations.

• Choose data types in each connector deliberately, as in lesson 3, so unneeded tables never arrive.
• Use event sets and XPath queries for Windows, and facilities and severities for Syslog, as in lesson 4.
• Send high-volume data that no rule uses to the lake only, so it is still searchable.
• Drop records or columns with DCR transformations, only with a recorded reason, because dropped data is gone.

Ama also noticed the AdditionalExtensions column in CommonSecurityLog, which holds the extra CEF fields that a vendor sends beyond the standard ones. Microsoft warns that it can be large and variable in volume. A week of trial data showed it was a quarter of the firewall volume and contained mostly debug information, so the team planned a transformation to drop that column from the store firewalls' records, with Nadia's approval, after checking that none of the vendor's rule templates used it.

Onboarding the lake

The lake is enabled at tenant level from the Defender portal, by someone with the right Microsoft Entra and Azure roles, and it must be in a supported region. Once onboarded, the old long-term retention and auxiliary log meters for that workspace are replaced by the lake meters. Because this changes billing and tenant-level settings, Nadia scheduled it as a change for December 7, 2026, after the free trial ended and real ingestion figures were known, rather than doing it during the sprint.

Retention also has a privacy side. Keeping personal data longer than needed is a risk in itself, so the two-year and one-year periods are written in the security workspace's record of processing with their reasons: investigation needs and the insurer's expectations. If those reasons change, the periods should change too.