Communicating with customers and leadership during an incident

In every incident there are two jobs. One is fixing the problem. The other is telling the people who depend on the facility what is happening, so that they can protect their own services, make their own decisions and trust the people running the building. Tenants judge an operator at least as much by how it communicates in an incident as by the incident itself. A short outage handled with prompt, honest, regular updates can strengthen a relationship; a smaller one handled with silence and vague answers can end it. This lesson is about the communication job: who must be told, when, what a good message says and what it must never say.

Who needs to know

The communications lead of Lesson 7, often the shift lead himself in a small incident, works from a notification list in each EOP and in the incident procedure. The list depends on what is affected.

• Affected tenants, through the contacts their contracts name, usually an on-call engineer and a manager. Corvid AI's contract, for example, requires notification of any severity 1 or 2 incident in Hall D within 15 minutes of declaration.
• Tenants who are not affected but could be, if the incident threatens shared systems such as a generator block or a chilled-water plant.
• Stonebridge leadership: the site manager, and for severity 1 the operations director and, depending on the impact, the account managers who look after the tenants commercially.
• Internal teams: security, the network operations team, other shifts and the campus maintenance planner.
• Outside parties when the situation requires: the fire department, the utility, the environmental regulator if a spill reaches drains or soil, the police for a security incident, and insurers.

The notification list is not a place to improvise during the event. The contacts and their order are set in advance, tested in drills and kept current, so that the communications lead makes calls rather than looking up numbers. In the drill of Lesson 6, the missing tenant numbers cost six minutes; after the drill they were added to the EOP.

When to tell them

Timing has two parts: the first notice and the rhythm of updates. Stonebridge's standard, written into its tenant contracts, is a first notice within 15 minutes of declaring a severity 1 or 2 incident and within an hour for severity 3 incidents that affect a tenant. Updates follow on a fixed rhythm, every 30 minutes for severity 1 and every 60 minutes for severity 2, until the incident is resolved, whether or not there is anything new to say. A final notice confirms resolution, and a written report follows.

The fixed rhythm matters more than it seems. People waiting for news assume the worst when none comes. An update that says the situation is unchanged, the electrician is twenty minutes away and the next update will be at 15:00 is far better than silence, because it tells the tenant that someone is in control and when to look again. It also protects the incident team: tenants who know when the next update is coming do not keep calling the operations center for news.

Speed of the first notice matters too. Tenants often see problems on their own monitoring before the operator tells them, and the first call they get should not come from their own engineers asking what is happening. Stonebridge's rule is that the first notice goes out when the incident is declared and its effect is roughly known, even if almost nothing is known about the cause. A first notice can be very short.

What a good message says

Every incident message at Stonebridge has the same four parts, in this order.

• What is happening: the facts, in plain language, without guessing at causes.
• What it means for the reader: which services, halls, cabinets or circuits are affected and how.
• What is being done: the actions under way and any action the reader may want to take.
• When the next update will come, and who to contact in the meantime.

The parts are simple, but writing them under pressure is not, so Stonebridge keeps templates for each type of incident. The communications lead fills in the facts and sends it. Templates also protect against the most common mistakes, by leaving out the places where they creep in.

# Fill an incident notification from a template, so every message has the same four parts
template = (
    "INCIDENT {ref} - severity {sev} - update {n}\n"
    "What is happening: {what}\n"
    "Effect on you: {effect}\n"
    "What we are doing: {doing}\n"
    "Next update: {next_update}, or sooner if anything changes. Contact: operations center, {phone}."
)
facts = {
    "ref": "IAD3-2027-0219", "sev": 2, "n": 1,
    "what": "At 14:12 the main breaker of power panel RPP-A2-08 opened. The A-side feed to 14 cabinets in cage A-26 is off.",
    "effect": "All 14 cabinets are running on their B-side feed, which is normal and within capacity. "
              "One single-corded device in A-26-05 has lost power.",
    "doing": "An electrician is on the way to find the cause before the breaker is closed. All planned work on the B side is stopped.",
    "next_update": "15:15",
    "phone": "the number on your access card",
}
msg = template.format(**facts)
print(msg)
missing = [k for k in ("what", "effect", "doing", "next_update") if not facts.get(k)]
print()
print("ready to send" if not missing else f"missing parts: {missing}")

INCIDENT IAD3-2027-0219 - severity 2 - update 1
What is happening: At 14:12 the main breaker of power panel RPP-A2-08 opened. The A-side feed to 14 cabinets in cage A-26 is off.
Effect on you: All 14 cabinets are running on their B-side feed, which is normal and within capacity. One single-corded device in A-26-05 has lost power.
What we are doing: An electrician is on the way to find the cause before the breaker is closed. All planned work on the B side is stopped.
Next update: 15:15, or sooner if anything changes. Contact: operations center, the number on your access card.

ready to send

This was Dmitri's first notice to Halvorsen Logistics in the breaker incident of Lesson 7, sent at 14:25, ten minutes after declaration. It says nothing about the cause, which was not yet known, and makes no promise about when the A side would return. It gives the customer what they need: their equipment is running on one feed, one device is down, someone is dealing with it, and they will hear again within the hour.

What a message must not say

Some things do not belong in incident messages, however tempting they are.

• Guesses about the cause. Early theories are often wrong, and a wrong cause in writing is hard to take back. Say what is known; the cause comes in the report.
• Blame. Never name an individual, a vendor or a tenant as at fault during an incident, even when it seems obvious. In the breaker incident the damaged cord belonged to the customer's own power strip, but saying so in an update would have been premature, unkind and, as it turned out, not the whole story.
• Promises without a basis. A time for recovery should be given only when the people doing the work can support it, and then as an estimate. Missing a promised time is worse than not giving one.
• Reassurance that is not true. Do not say the load is protected if it is running on a single path; say it is running on the B side, which is normal, and that redundancy is not available until the A side is restored.
• Information about other tenants. A tenant is told about its own equipment and about shared systems, never about another tenant's equipment, names or problems.
• Sensitive security details. A security incident is described in terms of its effect and the response, not how access was gained, which would help the next intruder.

Jargon also gets in the way. The people reading an incident message may be managers, not engineers, and they may be reading it on a phone at 03:00. Static bypass and RPP mean little to them. Words such as running on backup power, cooling capacity is reduced or one of two power feeds is off work for everyone, with the technical detail available to engineers who ask.

One voice

During an incident, only the communications lead and the incident commander speak for Stonebridge to tenants and leadership. Technicians on the floor will often meet tenant staff, and Corvid AI's site engineers in particular spend their days in Hall D. When a tenant asks a technician during an incident what is going on, the right answer is friendly and short: here is what I can see, the shift lead is running this and the updates are coming from the operations center, here is the number. The reason is not secrecy. A technician sees one part of the incident; a remark about what he thinks caused it, or how long it will take, can reach the tenant's management as the operator's official view and contradict what the commander has said.

The same rule applies to social media, the press and anyone outside the company. Stonebridge's policy is that nobody on site comments on an incident outside the company, and any press enquiry is passed to the communications team at head office. Large outages at well-known data centers do get reported, and a careless post by an employee can become the headline.

Talking to leadership

Leadership needs different things from tenants. The site manager and the operations director need to understand the risk, the decisions being made and any decision that needs their authority: approving an emergency purchase, accepting a risk, deciding whether to bring in more people, telling a large tenant's executives. A leadership update is short and structured: the current state, the risk to the load, what has been decided, what needs their decision, and when they will hear next. Anjali's standing request to her shift leads is: tell me early, tell me straight, and tell me what you need.

The bridge and the channel

In a large incident, coordination moves to two places: a conference call, called the bridge, which the commander runs and the key people join, and a written channel in the operations team's messaging system where the scribe posts the timeline and decisions. Tenants with large footprints, such as Corvid AI and the cloud provider, may be invited to a separate customer bridge run by the communications lead. Keeping the internal bridge and the customer bridge separate lets the team talk freely about options and uncertainties while the customers receive a clear, agreed picture.

After the incident

Communication does not end with the closing notice. Stonebridge sends affected tenants a preliminary incident report within one business day of a severity 1 or 2 incident, summarizing the timeline, the impact and the immediate actions, and stating that the root cause analysis is under way. A final report follows within ten business days, with the root cause, the contributing factors and the corrective actions, as Lessons 9 and 10 describe. Large tenants often ask for a meeting to go through it. These reports are written carefully and reviewed by the site manager, because they become part of the record of the relationship and sometimes part of a contract discussion about service credits.

# Update schedule for an incident: first notice and regular updates until resolution
from datetime import datetime, timedelta
declared = datetime(2027, 4, 11, 2, 52)
resolved = datetime(2027, 4, 11, 6, 52)
rules = {1: (15, 30), 2: (15, 60), 3: (60, 120)}   # severity: (first notice within, update every) minutes
sev = 1
first, every = rules[sev]
t = declared + timedelta(minutes=first)
times = [("first notice by", t)]
while t + timedelta(minutes=every) < resolved:
    t += timedelta(minutes=every)
    times.append(("update", t))
times.append(("resolution notice", resolved))
for what, when in times:
    print(f"{when:%H:%M}  {what}")
print(f"{len(times)} messages; preliminary report due next business day, final report within 10 business days")

03:07  first notice by
03:37  update
04:07  update
04:37  update
05:07  update
05:37  update
06:07  update
06:37  update
06:52  resolution notice
9 messages; preliminary report due next business day, final report within 10 business days

For the Hall A leak, the rhythm means nine messages to tenants between the first notice at 03:07 and the resolution notice. Writing them is a real job, which is why the communications lead in a severity 1 incident is a separate person, not the commander.