Incident response: detect, contain, communicate

Most of the time an alarm is handled by its alarm response procedure and closed within the hour: a filter needs changing, a sensor has drifted, a door was left open. Sometimes an event is bigger. Equipment fails in a way that affects tenants, several things go wrong at once, the cause is not obvious or the situation is getting worse. Then Stonebridge declares an incident and runs it in a structured way. This lesson is about that structure: what an incident is, how it is declared and graded, who does what and how the response moves from the first alarm to a stable, recovered facility. Communication, which is half the work, has Lesson 8 to itself.

What counts as an incident

Stonebridge's definition starts from the one used in IT service management frameworks such as ITIL 4, where an incident is an unplanned interruption to a service or a reduction in its quality, and adds any event that could cause one if nothing is done. For a colocation operator the service is power, cooling, connectivity, security and space for tenants' equipment. So an incident is not only an outage. The loss of redundancy that Lesson 1 described is an incident, even if no tenant notices it, because the next failure could drop the load. A fire alarm in a data hall is an incident even if it turns out to be dust. A security breach of a cage is an incident even if nothing was touched.

Not every alarm is an incident, and not every incident is an emergency. The shift lead decides whether an event becomes an incident, and grades it by its severity. Stonebridge uses four levels.

Severity can change as the incident develops. A chilled-water drip under a CRAH begins as severity 3; if it reaches a floor distribution unit it becomes severity 1. Raising severity early and lowering it later is always better than the reverse, and nobody is criticized at Stonebridge for raising an incident that turns out smaller than feared.

The phases of a response

Every incident moves through the same phases, though in a fast-moving event they overlap.

• Detect: something shows that a problem exists, usually an alarm, sometimes a tenant's call, a technician on a round or a smell.
• Acknowledge and assess: a person takes ownership, confirms the problem is real and gets a first picture of what is affected.
• Declare: the shift lead decides it is an incident, sets its severity and names the incident commander.
• Contain: stop it getting worse and protect people and the load, using the alarm response and emergency procedures of Lesson 6.
• Recover: restore the failed service or redundancy, by repair, replacement or a temporary arrangement.
• Close: confirm everything is back to normal, hand any follow-up to problem management and record the incident fully.

The order of the middle phases matters. Containment comes before cause. A technician's instinct is to find out why something happened, but during the incident the first job is to stop the damage. The question of why is answered later, carefully, in the root cause analysis of Lesson 9, using evidence that the response has been careful to preserve.

Roles

Small incidents need one person. Larger ones fail when everybody does everything, so Stonebridge assigns roles, borrowing the idea from the incident command systems used by fire and emergency services. The roles can be held by different people or combined, depending on the size of the incident.

The most important rule about roles is that there is always exactly one incident commander, and everyone knows who it is. Commanders can hand over, for example when the site manager arrives and takes command from the shift lead, but the handover is said aloud and recorded: Anjali has command from 03:40. A second rule is that the commander in a serious incident keeps their hands free. Dmitri likes to say that the moment the commander picks up a tool, nobody is commanding.

A severity 2 incident

Jamal sees the structure work on a weekday afternoon in his third week on the rotation.

Containment: what not to do

The example shows the most important containment habit: do not restore something until you know why it failed. Resetting a tripped breaker, restarting a failed unit or clearing a latched alarm can work, and when it works nobody remembers the risk. When it does not, it turns a severity 2 into a severity 1. Stonebridge's ARPs for protective devices all start the same way: leave it as it is, confirm the remaining path, call the qualified person.

A second habit is to preserve evidence. Before anything is changed, photographs are taken of the equipment's state, the positions of breakers and valves, the alarm screens and any visible damage. Event logs on UPS systems, breaker trip units and controllers are saved, because some of them overwrite their oldest entries. Failed parts are kept, tagged with the incident number, not thrown away. The root cause analysis depends on all of this.

A third is to protect what is still working. During the breaker incident Dmitri stopped planned work on the B side, because the B side was now the only path for 14 cabinets. Freezing changes on the path that is carrying the load is a standing rule during any severity 1 or 2 incident.

Escalation

Escalation means bringing in more help or more authority. Stonebridge's escalation rules are written into each EOP and into the severity table: severity 1 calls the site manager and the operations director immediately, severity 2 calls the site manager within fifteen minutes, and any incident that is not contained within an hour, or not recovered within four, is escalated one level higher. Technical escalation runs in parallel: the vendors on emergency contracts, with their response times from Lesson 3, the electrical contractor, the controls engineers. A commander who escalates too early wastes some people's evening; one who escalates too late can lose the load. Stonebridge's culture, like most in the industry, prefers the first mistake.

Measuring the response

The scribe's timeline makes it possible to measure each incident's response. Four intervals are tracked: time to detect, from the start of the problem to its first detection; time to acknowledge, from detection to a person taking ownership; time to contain, from detection to the problem stopping getting worse; and time to recover, from detection to the service or redundancy being restored. The start of the problem is not always known at once; it is often found later from logs.

# Incident timeline and response metrics from the scribe's log
from datetime import datetime
fmt = "%H:%M"
log = [
    ("14:11", "start", "fault on branch circuit to A-26-11 (from trip unit log)"),
    ("14:12", "detect", "RPP-A2-08 main breaker trip alarm"),
    ("14:12", "ack", "monitoring desk acknowledges, calls shift lead"),
    ("14:15", "declare", "severity 2 declared, Dmitri commands"),
    ("14:22", "contain", "B side confirmed stable, B-side work frozen"),
    ("15:05", "info", "electrician on site"),
    ("15:40", "recover", "A side restored to 13 cabinets and storage controller"),
    ("16:30", "close", "incident closed, problem record opened"),
]
t = {k: datetime.strptime(ts, fmt) for ts, k, _ in log if k != "info"}
mins = lambda a, b: int((t[b] - t[a]).total_seconds() // 60)
for ts, k, what in log:
    print(f"{ts}  {k:8} {what}")
print()
print(f"time to detect      {mins('start', 'detect'):3} min")
print(f"time to acknowledge {mins('detect', 'ack'):3} min")
print(f"time to contain     {mins('detect', 'contain'):3} min")
print(f"time to recover     {mins('detect', 'recover'):3} min")

14:11  start    fault on branch circuit to A-26-11 (from trip unit log)
14:12  detect   RPP-A2-08 main breaker trip alarm
14:12  ack      monitoring desk acknowledges, calls shift lead
14:15  declare  severity 2 declared, Dmitri commands
14:22  contain  B side confirmed stable, B-side work frozen
15:05  info     electrician on site
15:40  recover  A side restored to 13 cabinets and storage controller
16:30  close    incident closed, problem record opened

time to detect        1 min
time to acknowledge   0 min
time to contain      10 min
time to recover      88 min

Each number points to a different improvement. Time to detect depends on monitoring and rounds; time to acknowledge on staffing and alarm handling; time to contain on procedures and training; time to recover on spares, vendors and the design of the plant. In this incident the longest interval was the electrician's travel time, and the problem review later asked whether a qualified electrician should be on the campus during the day shift. The timing figures are kept for every incident, and their trends are reviewed every quarter.

Closing an incident

An incident is closed when the service and any lost redundancy are restored and stable, when the tenants who were affected agree, and when the record is complete: timeline, actions, communications, evidence collected and what remains to be done. Closing the incident does not close the question of why it happened. That moves to a problem record; in ITIL terms a problem is the cause, or potential cause, of one or more incidents, and the record tracks the investigation, which leads on to the root cause analysis and corrective actions of Lessons 9 and 10. Jamal learns to think of the incident as the fire and the problem as the fire investigation: the first is about speed, the second about care.