Disaster recovery sites, plans and testing

A disaster recovery plan is a promise: if the primary facility or platform is lost, these systems will be running again within these times. Keeping that promise depends on three things. There must be somewhere to recover to, with the capacity and connectivity required. There must be a plan that people can follow under stress, with clear roles and up-to-date procedures. And the plan must have been tested often enough that the organization knows it works and its people know what to do. This lesson covers those three things, along with the business continuity exercises that bring the wider organization into the picture.

The CISSP outline covers these topics in sub-domains 7.10 (recovery site strategies and multiple processing sites), 7.11 (implement disaster recovery processes: response, personnel, communications, assessment, restoration, training and awareness, lessons learned), 7.12 (test disaster recovery plans: read-through or tabletop, walkthrough, simulation, parallel and full interruption) and 7.13 (participate in business continuity planning and exercises). NIST SP 800-34 Revision 1 and ISO 22301 remain the main references.

Recovery site options

Recovery sites are traditionally described by how ready they are. A hot site is a fully equipped facility with current systems and data, able to take over within minutes to hours. A warm site has facilities, power, network and some hardware, but systems need configuration and data restoration, taking hours to days. A cold site provides space, power and cooling but no ready equipment, so recovery takes days to weeks. A mirrored or active-active arrangement runs production in two sites at once, so either can carry the load with little or no interruption. Other options include mobile sites delivered in trailers or containers, reciprocal agreements with another organization to share facilities, and service bureaus or recovery providers that offer shared facilities on subscription.

Cloud platforms have changed these economics. Approaches such as backup and restore to the cloud, pilot light (keeping core data replicated and a minimal environment ready to scale up), warm standby (a scaled-down copy running continuously) and multi-region active-active let organizations buy short RTOs without owning a second data centre. They introduce their own considerations: infrastructure as code and automation must be maintained so the environment can be built on demand; cloud region capacity may be constrained during a widespread event; identity and network dependencies must also be recoverable; and data residency requirements, such as keeping certain Canadian public sector or health data in Canada, may limit which regions can be used.

Geographic separation matters for any site. The recovery location should be far enough from the primary that a single regional event, such as a flood, wildfire, ice storm, earthquake or extended power outage, is unlikely to affect both, but close enough for staff to reach if they must travel, and for replication latency to be acceptable. Shared dependencies, such as the same telecommunications provider's central office or the same electrical substation, can defeat apparent separation.

# Choose the cheapest recovery option that meets each tier's RTO (annual cost, synthetic)
options = [
    ('cold site', 120, 30000),
    ('cloud backup and restore', 24, 45000),
    ('warm site', 12, 140000),
    ('cloud pilot light', 4, 90000),
    ('cloud warm standby', 1, 210000),
    ('hot site', 1, 420000),
    ('active-active', 0.1, 650000),
]
tiers = {'Tier 0 foundation': 2, 'Tier 1 ordering and dispatch': 6, 'Tier 2 support and payroll': 48, 'Tier 3 deferrable': 168}
total = 0
for tier, rto in tiers.items():
    ok = [o for o in options if o[1] <= rto]
    name, opt_rto, cost = min(ok, key=lambda o: o[2])
    total += cost
    print(f'{tier:30s} RTO {rto:>4} h -> {name:25s} (RTO {opt_rto} h, ${cost:,}/yr)')
print(f'Total annual recovery cost: ${total:,}')

Tier 0 foundation              RTO    2 h -> cloud warm standby        (RTO 1 h, $210,000/yr)
Tier 1 ordering and dispatch   RTO    6 h -> cloud pilot light         (RTO 4 h, $90,000/yr)
Tier 2 support and payroll     RTO   48 h -> cloud backup and restore  (RTO 24 h, $45,000/yr)
Tier 3 deferrable              RTO  168 h -> cold site                 (RTO 120 h, $30,000/yr)
Total annual recovery cost: $375,000

In this synthetic comparison, cloud warm standby meets the foundation tier's two-hour RTO at half the cost of a hot site, and cloud pilot light meets Tier 1's six hours far more cheaply than a warm site. The point is not that cloud always wins, but that recovery cost should be decided tier by tier from the BIA, rather than by picking one site type for everything. A real comparison would include staff, licences, network and the cost of testing.

The disaster recovery plan

A DR plan must be usable by people who are tired, stressed and possibly not the usual system experts. NIST SP 800-34 suggests a structure with supporting information (purpose, scope, assumptions and concept of operations), an activation and notification phase, a recovery phase, a reconstitution phase and appendices with contact lists, vendor information, procedures and system details. The plan must be available when primary systems are not: printed copies, copies on secured mobile devices and copies at the recovery site are all common.

Declaring a disaster is a significant decision: it may trigger costs under recovery contracts, move staff and change how customers are served. The plan should name who can declare (usually a small group of senior managers, with alternates) and give criteria, such as an expected outage longer than a system's RTO. Delay in declaring is a common failure: teams spend hours trying to fix the primary site, then start recovery too late to meet objectives.

Returning to the primary site, sometimes called failback, deserves as much planning as failover. It must be scheduled to minimize disruption, data changed during the recovery period must be synchronized back, and the primary must be verified before traffic moves. Many organizations rehearse failover but never failback, and discover the difficulty only after a real event.

Personnel, communications and assessment

People are the most important part of a DR plan and the most often overlooked. The plan should identify primary and alternate staff for each role, recognizing that some people may be unavailable because they are affected by the same event. Staff need safe travel and lodging if they must work at the recovery site, and rotations for extended operations. Their families' safety comes first; personnel safety and emergency management are discussed in Lesson 12, and occupational health and safety obligations are covered in the HSE courses.

Communications in a disaster follow the same principles as in incident response: pre-approved templates, authorized spokespeople, alternate channels that do not depend on the affected systems and regular, date-stamped updates. Damage assessment, early in the activation phase, determines what has been lost, how long the primary will be unavailable and therefore whether to declare. Throughout, the team should log decisions and actions; that record supports insurance claims, regulatory questions and the lessons-learned review.

Testing the plan

Testing proves that the plan works and trains the people who must carry it out. The CISSP outline names five types of test, in increasing order of realism, cost and risk. A read-through or checklist review has plan owners review the plan for accuracy and completeness. A walkthrough or tabletop exercise has the team talk through a scenario step by step. A simulation rehearses the response to a realistic scenario, sometimes with actions performed, but without actually failing over production. A parallel test brings up systems at the recovery site and processes real or copied workloads there while production continues at the primary site. A full interruption test shuts down the primary and moves production to the recovery site; it gives the most assurance but carries real business risk and needs senior approval.

Every test should have objectives set in advance (for example, restore the order database within six hours with data no older than 15 minutes), measured results, and a report with findings and corrective actions tracked to closure. Tests that always succeed may be too easy; the purpose is to find problems while the cost of finding them is low.

# Summarize DR test results against objectives (synthetic)
tests = [
    # system, test type, RTO target h, achieved h, RPO target min, achieved min
    ('order-db', 'parallel', 6, 4.5, 15, 6),
    ('web-store', 'parallel', 6, 7.2, 15, 6),
    ('identity', 'simulation', 2, 1.6, 5, 2),
    ('payroll', 'tabletop', 48, None, 1440, None),
    ('dispatch-app', 'parallel', 6, 5.8, 60, 75),
]
met = 0
measured = 0
for sysname, kind, rto_t, rto_a, rpo_t, rpo_a in tests:
    if rto_a is None:
        print(f'{sysname:13s} {kind:10s} not measured (discussion-based test)')
        continue
    measured += 1
    ok = rto_a <= rto_t and rpo_a <= rpo_t
    met += ok
    note = 'met' if ok else 'ERROR ' + ', '.join(x for x, bad in [('RTO missed', rto_a > rto_t), ('RPO missed', rpo_a > rpo_t)] if bad)
    print(f'{sysname:13s} {kind:10s} RTO {rto_a:4.1f}/{rto_t} h  RPO {rpo_a:3d}/{rpo_t} min  {note}')
print(f'{met} of {measured} measured tests met both objectives')

order-db      parallel   RTO  4.5/6 h  RPO   6/15 min  met
web-store     parallel   RTO  7.2/6 h  RPO   6/15 min  ERROR RTO missed
identity      simulation RTO  1.6/2 h  RPO   2/5 min  met
payroll       tabletop   not measured (discussion-based test)
dispatch-app  parallel   RTO  5.8/6 h  RPO  75/60 min  ERROR RPO missed
2 of 4 measured tests met both objectives

Two systems missed objectives. The web store took longer than its RTO because, as in Lesson 9's example, it waited for a dependency; and the dispatch application lost more data than its RPO allowed because its replication interval had drifted from the design. Both become tracked corrective actions, and both would have been unpleasant surprises in a real disaster.

Business continuity exercises

Business continuity exercises extend testing beyond IT. They examine whether business units can operate with workarounds, such as paper forms, alternate suppliers or staff working from other locations, while systems are restored, and whether leaders can make decisions and communicate effectively. ISO 22301 requires organizations to exercise and test their continuity procedures at planned intervals and after significant changes, and to review the results. A well-designed exercise program builds over time: from discussion-based tabletops, to functional exercises that test specific capabilities, to full-scale exercises involving multiple teams and sometimes external partners such as suppliers, emergency services or regulators.

Training and awareness complete the picture. Everyone should know the basics, such as where to go and how to receive information during an emergency, and people with continuity or DR roles need deeper training and practice. Plans must be maintained: reviewed at least annually, after each test or real event, and whenever systems, staff, suppliers or business priorities change. Version control and distribution lists ensure that the copy people reach for in a crisis is the current one.