Incident management II: resolution, major incidents and swarming

Lesson 3 took an incident from detection to a clear priority. This lesson follows it to resolution. It covers escalation, diagnosis and resolution, closure, the special handling of major incidents, and the shift from rigid tiers toward swarming. It also measures incident performance with mean time to restore service, mean time between failures and availability, and explains where incident management meets information security and the legal duty to report breaches.

Escalation

When the service desk cannot resolve an incident, it escalates. ITIL traditionally distinguishes two kinds. Functional escalation passes the incident to a team with more specialized skills or access, such as the network team or an application support group. Hierarchical escalation informs or involves more senior managers, usually because the incident is serious, a target is at risk or a decision is needed that the current team cannot make, such as approving overtime or invoking a supplier's premium support.

Escalation rules should be explicit. Typical triggers are a time limit at the service desk (for example, escalate if not resolved within 15 minutes for P2 and above), a need for skills or access the desk lacks, or a target that is about to be breached. Clear ownership is just as important: the incident should have one owner, often the service desk, who stays accountable for coordination and communication even while specialists do the technical work. Incidents that bounce between teams with nobody owning them are a classic failure.

Diagnosis and resolution

Diagnosis means finding out enough about what is wrong to restore service. It does not necessarily mean finding the root cause, which is problem management's job. Resolvers gather information from monitoring, logs, recent changes and the configuration management system; reproduce the symptoms where possible; and test hypotheses one at a time. Good diagnosis is methodical. Changing several things at once may restore service but leaves nobody sure what worked, which makes the next occurrence harder.

Resolution restores service. It may be a permanent fix, but often it is a workaround: restarting a service, failing over to a standby system, rolling back a recent change or giving users an alternative way to work. ITIL is clear that a workaround which restores service is a successful resolution of the incident, even if the underlying cause remains. Recovery then confirms that service is fully back: data synchronized, queues cleared and dependent services working.

Closure

Closure confirms that the user is satisfied and the record is complete. Many organizations resolve an incident first and close it a few days later if the user does not object, which avoids reopening a closed record when the fix does not hold. At closure, the resolver records the actual cause category, the resolution, whether a workaround was used, and whether a problem record should be raised. A short satisfaction survey may be sent. Closure data feeds problem management, knowledge management and continual improvement, so quality matters.

Measuring incident performance

Several measures describe how incident management is performing. Mean time to restore service (MTTR, sometimes called mean time to repair or recover) is the average time from detection to restoration. Mean time between failures (MTBF) is the average time a service runs between incidents. Availability is the share of agreed service time during which the service is usable. A standard relationship links them for a repairable service: availability equals MTBF divided by the sum of MTBF and MTTR.

The formula shows two ways to improve availability: fail less often (raise MTBF, mostly the work of problem management, design and change) or recover faster (lower MTTR, mostly the work of incident management, monitoring and knowledge). Incident management's distinctive contribution is the second.

Averages can hide a great deal. A mean of 2 hours might come from most incidents taking 30 minutes and a few taking 20 hours. Percentiles, such as the time within which 90 % of incidents are restored, often describe the user experience better. The program below generates a synthetic month of incidents with a seeded random generator, computes MTTR, MTBF, availability and percentiles, and shows how much a few long incidents affect the mean.

import random, statistics

random.seed(140)
HOURS = 30 * 24                       # one 30-day month
# synthetic incidents: most restore quickly, a few take much longer (lognormal)
restore_min = [round(random.lognormvariate(3.6, 0.9)) for _ in range(18)]
restore_min.sort()

down_h = sum(restore_min) / 60
mttr_h = statistics.mean(restore_min) / 60
mtbf_h = (HOURS - down_h) / len(restore_min)
avail = mtbf_h / (mtbf_h + mttr_h)

def pct(data, p):
    k = (len(data) - 1) * p
    f, c = int(k), min(int(k) + 1, len(data) - 1)
    return data[f] + (data[c] - data[f]) * (k - f)

print("restore times (min):", restore_min)
print(f"incidents: {len(restore_min)}   total downtime: {down_h:.1f} h")
print(f"MTTR: {mttr_h*60:.0f} min   median: {statistics.median(restore_min):.0f} min   90th percentile: {pct(restore_min, 0.9):.0f} min")
print(f"MTBF: {mtbf_h:.1f} h   availability: {avail:.3%}")
print(f"check: (720 - downtime) / 720 = {(HOURS - down_h) / HOURS:.3%}")

restore times (min): [8, 17, 22, 23, 27, 32, 35, 36, 37, 56, 58, 81, 84, 93, 98, 131, 264, 336]
incidents: 18   total downtime: 24.0 h
MTTR: 80 min   median: 46 min   90th percentile: 171 min
MTBF: 38.7 h   availability: 96.671%
check: (720 - downtime) / 720 = 96.671%

The mean restoration time sits well above the median because a handful of long incidents pull it up. Reporting both, plus a high percentile, tells service owners whether they have a general speed problem or a few incidents that need special attention. The final line confirms that the MTBF and MTTR formula gives the same availability as the simple downtime calculation when MTBF is measured as uptime per incident.

Major incidents

A major incident is an incident with significant business impact that requires a response beyond the normal process. Each organization defines its own criteria, typically a P1 on a critical service, a widespread outage, a threat to safety or a serious reputational or financial risk. Major incidents get a separate procedure because normal queues, handoffs and update intervals are too slow.

A typical major incident procedure names a major incident manager who coordinates the response and does not do the technical work; opens a bridge call or dedicated chat channel; brings the right technical teams and suppliers together at once; assigns a communications lead to update users, executives and the service desk at fixed intervals; tracks decisions and actions in a timeline; and agrees in advance who may approve emergency changes. When service is restored, the incident is not finished until a post-incident review has been held.

Post-incident reviews

A post-incident review, also called a retrospective or post-mortem, examines what happened, how it was detected, how the response went and what should change. The best reviews are blameless: they assume people acted reasonably with the information they had and look for system weaknesses rather than individuals to punish. Blame drives people to hide mistakes, which destroys the information the organization needs to learn. Typical outputs are a timeline, contributing factors, what went well, what could be improved, and actions with owners and dates. Actions to remove causes go to problem management; actions to improve the response go to incident management itself.

From tiers to swarming

Traditional support uses tiers. The service desk (often called tier 1 or first line) handles what it can and escalates the rest to tier 2 specialists, who escalate further to tier 3 experts or suppliers. Tiers are simple to manage but create queues and handoffs, and each handoff loses information and adds delay. Experts at higher tiers rarely see the easy cases, so knowledge does not flow back down.

Swarming is an alternative. Instead of passing an incident up a ladder, the first person who cannot resolve it pulls in the right people immediately, through a chat channel or a short meeting, and they work it together until it is resolved. Some organizations run regular swarms where experts are available at set times; others use severity-based swarms. Swarming reduces handoffs, spreads knowledge and often cuts resolution time. It also needs a culture of helpfulness, tools that make experts reachable and some protection for experts' time so they are not constantly interrupted. Many organizations combine both models: tiers for routine work, swarming for complex or high-impact incidents.

Incident management and information security

Lesson 1 distinguished ITIL incidents from security incidents. In practice the two meet constantly. The service desk is often where users first report suspicious activity, and a ransomware attack is both a service outage and a security incident. Organizations need a clear handover: the service desk recognizes possible security events, follows a defined procedure, preserves evidence (for example by not wiping an affected laptop) and passes the case to the security team, which leads containment, eradication and recovery. Incident management continues to handle the service impact and communication with users. Mixing the two without a plan can destroy evidence or delay legal notifications.

Security incidents that involve personal information can trigger legal reporting duties. These are summarized below at a level useful for service management staff; ITIL-141 and the CISSP courses go further.

For incident management, the practical lessons are to recognize when an incident may involve personal information or a material security event, escalate to the right people quickly, record times carefully because deadlines run from discovery or determination, and avoid making public statements that have not been approved.