Incident management II: containment, recovery, communication and lessons learned

Once an incident has been detected and its severity set, the organization faces a series of decisions under pressure: how to limit the damage, when and how to remove the cause, how quickly to restore service, what to tell staff, customers, partners and regulators, and how to learn from what happened. These decisions are rarely purely technical. Isolating a system may stop an intruder but also halt production; restoring quickly may destroy evidence; communicating early builds trust but risks stating facts that later prove wrong. This lesson treats each stage as a management decision with trade-offs that the incident manager must surface to the right people.

In NIST SP 800-61r3 these activities fall under the Respond and Recover functions of CSF 2.0, with lessons learned feeding improvement across every function. In ISO/IEC 27035-1 they are the respond and learn lessons phases. The CISSP outline calls them response, mitigation, reporting, recovery, remediation and lessons learned. Legal and regulatory reporting obligations are important enough to have their own lesson (Lesson 8); here they appear only as one of several communication streams.

Containment decisions

Containment limits the spread and impact of an incident while the organization works out what happened. It buys time. Typical containment options include isolating affected systems from the network, disabling compromised accounts or forcing credential resets, blocking malicious addresses or domains, suspending a vulnerable service, restricting data transfers, and increasing monitoring. Short-term containment is often followed by longer-term containment, such as moving services to clean infrastructure, while eradication is planned.

Choosing a containment strategy means weighing several factors that the plan should name in advance: potential damage and theft of resources; the need to preserve evidence; service availability and business impact; the time and resources needed; how effective the strategy is likely to be; and how long the solution will last. The decision to take a revenue-generating or safety-related system offline belongs to the business owner or executive sponsor, informed by security's assessment. The incident manager's job is to put the options and their consequences in front of that person quickly and clearly.

Timing matters. Acting on one compromised account while others remain unknown can alert an adversary, who may then cause more damage or hide. In serious incidents responders therefore try to understand scope before acting, then contain all known footholds together. Where an adversary is extorting the organization or data is actively leaving, the balance shifts toward immediate action. These are judgment calls that the plan cannot fully script, which is why practiced decision makers matter.

# Weigh containment options against agreed criteria (scores 1 = poor, 5 = good; synthetic)
criteria = {'stops_damage': 0.35, 'preserves_evidence': 0.15, 'keeps_service': 0.30, 'speed': 0.20}
options = {
    'isolate 14 file servers now':        {'stops_damage': 5, 'preserves_evidence': 3, 'keeps_service': 1, 'speed': 5},
    'block outbound to known addresses':  {'stops_damage': 3, 'preserves_evidence': 5, 'keeps_service': 5, 'speed': 5},
    'reset all admin credentials':        {'stops_damage': 4, 'preserves_evidence': 4, 'keeps_service': 3, 'speed': 3},
    'monitor for 24 h, then act':         {'stops_damage': 1, 'preserves_evidence': 5, 'keeps_service': 5, 'speed': 1},
}
ranked = sorted(options.items(), key=lambda kv: -sum(criteria[c] * kv[1][c] for c in criteria))
for name, s in ranked:
    total = sum(criteria[c] * s[c] for c in criteria)
    print(f'{total:4.2f}  {name}')
print('Note: scores inform the decision; the business owner approves service-affecting actions.')

4.30  block outbound to known addresses
3.50  isolate 14 file servers now
3.50  reset all admin credentials
2.80  monitor for 24 h, then act
Note: scores inform the decision; the business owner approves service-affecting actions.

The weighted model does not make the decision; it makes the trade-offs visible. In this example a combination is likely: block outbound traffic and reset administrator credentials at once, while the business decides whether to isolate file servers. Recording the options considered and the reasons for the chosen course is important later, when regulators, insurers or courts ask whether the organization acted reasonably.

Eradication and recovery at a process level

Eradication removes the cause of the incident: the adversary's access, any unauthorized software or accounts, and the vulnerability or misconfiguration that allowed entry. Recovery restores systems and services to normal operation and confirms that they are functioning securely. In practice, eradication and recovery are often combined by rebuilding affected systems from trusted images and restoring data from clean backups, rather than trying to clean systems in place, because it is difficult to be sure that a compromised system has been fully cleaned.

From a management perspective, recovery needs a plan with priorities. Systems should be restored in the order set by the business impact analysis (Lesson 9), with dependencies such as identity services, network infrastructure and backups themselves restored first. Each restored system should be checked against its baseline, patched, have credentials rotated and be monitored closely for signs of re-compromise. Backups must be checked to ensure they predate the compromise; restoring an infected backup reintroduces the problem. Recovery is also the moment to remediate root causes: if the incident began with an unpatched internet-facing system, the patching gap across all similar systems must be closed, not just on the one that was compromised.

The decision to declare an incident closed should also be deliberate. Criteria might include no new indicators for an agreed period, all affected systems rebuilt or verified, credentials rotated, root causes remediated or tracked, notifications completed and the post-incident review scheduled. Closing too early, under pressure to return to normal, is a common cause of repeat incidents.

Communication during an incident

Communication is one of the most consequential and least practiced parts of incident response. An incident typically has several audiences, each needing different information at a different pace: the response team, executives and the board, employees, customers, partners and suppliers, insurers, regulators, law enforcement and, for larger incidents, the media. The plan should name who is authorized to speak to each audience and require that external statements be reviewed by legal and communications staff.

Several practical rules apply. Use out-of-band communication when there is any chance the adversary can read corporate email or chat; many organizations keep a backup channel for exactly this reason. Separate facts from assumptions in every update, and date-stamp updates so readers know what was known when. Avoid speculating about attribution or about the volume of data affected until it is confirmed. Involve counsel early: in both Canada and the United States, communications prepared at the direction of counsel for the purpose of legal advice may be protected by solicitor-client or attorney-client privilege, but courts look closely at whether the main purpose was legal advice, and forensic reports prepared for business purposes have been ordered disclosed in US litigation.

US organizations should also remember that statements to individuals, state attorneys general and federal regulators must be accurate and consistent; public companies must also avoid misleading investors, which is why disclosure committees are often involved in incident communication.

Ransom and extortion decisions

Extortion incidents, in which an adversary encrypts systems or threatens to publish stolen data unless paid, force especially difficult decisions. The plan should set out in advance who decides whether payment will be considered and what must be checked first: whether backups allow recovery without payment, whether payment would be legal (for example, payments to sanctioned persons are prohibited under Canadian sanctions law and US OFAC regulations), what insurers require, and what law enforcement advises. Government agencies in both countries, including CCCS in Canada and the FBI, discourage payment because it funds criminal activity and does not guarantee recovery or deletion of data. Some US reporting rules, discussed in Lesson 8, require ransom payments to be reported. The best defense against having to make this decision is preparation: tested, isolated backups and a rehearsed recovery plan.

Lessons learned and continuous improvement

Every significant incident should end with a post-incident review, held soon enough that memories are fresh, typically within two weeks. The review should be blameless in tone: the goal is to understand how the organization's systems, processes and decisions allowed the incident to happen and to unfold as it did, not to find a person to punish. Useful questions include: What happened, and when, according to the confirmed timeline? What went well? What was harder than it should have been? Where did detection, escalation or communication break down? What would we do differently? What changes will we make, who owns them, and by when?

The output is an action plan whose items are tracked like any other risk treatment: updated playbooks, new detection rules, configuration changes, training, contract changes with suppliers, and sometimes investment requests. NIST SP 800-61r3 emphasizes that these improvements apply across all CSF functions; an incident may reveal a governance gap (unclear authority), an identification gap (unknown asset), a protection gap (missing multi-factor authentication), a detection gap or a recovery gap. Metrics from incidents, such as time to detect, time to contain and the proportion of action items closed on time, show whether the program is improving.

# Track post-incident action items and show on-time closure by function (synthetic)
items = [
    ('Govern', 'Clarify authority to isolate plant systems', True),
    ('Identify', 'Add 37 unmanaged servers to inventory', True),
    ('Protect', 'Enforce MFA on remote access for contractors', True),
    ('Protect', 'Remove local admin rights from finance laptops', False),
    ('Detect', 'Alert on mass file renames on file servers', True),
    ('Detect', 'Forward VPN logs to SIEM', False),
    ('Respond', 'Add out-of-band contact list to plan', True),
    ('Recover', 'Test restore of ERP database quarterly', True),
]
from collections import Counter
total, on_time = Counter(), Counter()
for fn, text, ok in items:
    total[fn] += 1
    on_time[fn] += ok
for fn in ['Govern', 'Identify', 'Protect', 'Detect', 'Respond', 'Recover']:
    print(f'{fn:9s} {on_time[fn]}/{total[fn]} closed on time')
print(f'Overall {sum(on_time.values())}/{len(items)} = {sum(on_time.values())/len(items):.0%}')

Govern    1/1 closed on time
Identify  1/1 closed on time
Protect   1/2 closed on time
Detect    1/2 closed on time
Respond   1/1 closed on time
Recover   1/1 closed on time
Overall 6/8 = 75%