A database operations runbook

On the last evening of the clinic's autumn series, Elena gives Walt a task instead of a lesson. "Your binder has served this company for seventeen years," she says. "By the end of 2027 it will describe servers that no longer exist. Write the new one: the runbook that someone else could pick up at three in the morning and use to keep the Azure estate healthy." This lesson follows Walt as he assembles the Harrow & Finch database operations runbook from the pieces built in this course: automated tasks and their owners, alerts and who answers them, recovery tiers and designs, backup and retention policies, and the tests that prove them. It is also a review of the course, and of the skills the DP-300 outline groups under automation and high availability and disaster recovery.

What a runbook is for

A runbook is a set of written procedures for operating a system: what happens routinely, what to do when something goes wrong, and who decides. A good runbook is written for the reader who knows the technology but not this estate, at the worst possible moment. It is short where it can be, exact where it must be, kept in a place everyone can reach even when Azure is having a bad day, and reviewed on a schedule. Walt's runbook has five parts: an estate map, routine operations, alerts and responses, recovery procedures, and testing and review. Each part links to the scripts and templates in the team's repository rather than copying them, so there is one version of each.

Part 1: the estate map

The first page lists every database with its owner, recovery tier, objectives, design and where its recovery procedure is. It is the page a responder reads first, because it answers the question that decides everything else: how much does this database matter, and what has been built for it?

Part 2: routine operations

The routine section lists every scheduled task with its frequency, tool, owner and the way a failure would be noticed, so that a missing run is as visible as a failed one. It brings together the Agent jobs and elastic jobs from lesson 1, the runbooks from lesson 2 and the alerts from lesson 3. Walt adds one daily human task: the on-call DBA reads the overnight summary at 07:30 and signs it off in the operations log. Automation reports, but a person confirms.

Part 3: alerts and responses

Every alert rule has a runbook entry with its meaning, its severity, its action group and the first three things to check. The entries are written as questions, because a responder at three in the morning thinks in questions. For example, for the alert "Replication lag above 30 seconds for 10 minutes, Finch Rewards": Is the primary running a large batch or rebuild? Is the secondary the same size as the primary? Is the replication link healthy? Each entry ends with when to escalate and to whom. Walt keeps severities honest: severity 0 means customers are affected now and someone must act within 15 minutes; severity 1 means they will be soon; anything else waits until working hours.

Part 4: recovery procedures

The recovery section starts with decision rules, because lesson 4 showed that deciding is the slowest step. The rules say who may declare a disaster for each tier, what evidence is enough, and what to do if the decision-maker cannot be reached within 10 minutes. For tier 1, the on-call DBA and the duty manager together may declare a disaster and force the StockLine failover if the primary region has been unavailable for 15 minutes and Azure Service Health confirms a regional incident; if the duty manager cannot be reached, the data platform lead may decide alone. Then each procedure lists its steps with commands, expected results and checks, for failover, restore and failback.

Failback deserves its own page. After a forced failover, the team will run in the secondary region for hours or days. The failback page lists the conditions for returning (the primary region declared healthy, the old primary resynchronized as secondary, a quiet period agreed with the business), the planned failover command, and the checks afterwards. It also says that failing back is optional: if the secondary region is serving well, the team may stay there until the next drill.

Part 5: testing and review

The last section holds the test calendar from lesson 11, the results of the most recent tests with their achieved RTO and RPO, and the list of improvements still open. It also sets a review date. Walt's runbook is reviewed every six months, after every real incident or production drill, and whenever the estate changes: a new database, a new tier, a change of region. Each review checks that the names, scripts and contacts are still right, which is where most runbooks quietly go wrong.

Python example

Walt wants the runbook's estate map to warn him automatically when a design no longer meets its tier. He writes a small check that compares each database's tested RTO and RPO with its objectives, and plans to run it after every test.

estate = [
    # name, RPO objective (min), RTO objective (min), last tested RPO (min), last tested RTO (min)
    ("StockLine",      5,   60,    0.2,  40),
    ("Finch Rewards", 60,  240,    0.1,  25),
    ("SalesHistory", 1440, 1440,  37,    66),
    ("Warehouse WMS", 15,  120,    0.5, 135),
]
for name, rpo, rto, t_rpo, t_rto in estate:
    problems = []
    if t_rpo > rpo: problems.append(f"RPO {t_rpo} > {rpo}")
    if t_rto > rto: problems.append(f"RTO {t_rto} > {rto}")
    status = "meets its tier" if not problems else "REVIEW: " + ", ".join(problems)
    print(f"{name:14s} {status}")

StockLine      meets its tier
Finch Rewards  meets its tier
SalesHistory   meets its tier
Warehouse WMS  REVIEW: RTO 135 > 120

The output flags the warehouse system, whose last test in the vendor's lab took 135 minutes against a 2-hour objective. That is exactly the kind of finding the runbook's review exists to catch, and Walt adds it to the open improvements: rehearse the availability group failover with the vendor's engineers before the warehouse goes live.

Looking back over the course

Walt's runbook is also a map of what this course covered. Automation of tasks: Agent jobs and elastic jobs for work inside databases, PowerShell, the CLI, runbooks and templates for work on resources, and alerts and action groups so that every task's failure is noticed. High availability and disaster recovery: recovery objectives agreed with the business, the built-in availability of each service tier and zone redundancy, geo-replication and failover groups across regions, availability groups, failover cluster instances and log shipping on virtual machines, point-in-time and long-term backups, and tests that turn a plan into evidence. MSAZ-281 covers the security, monitoring and tuning that keep the same databases safe and fast, and MSAZ-289 brings all three courses together for the exam.