Emergency operating procedures and alarm response procedures

Planned work at Stonebridge follows methods of procedure, written days or weeks ahead, reviewed and approved, as DCT-330 described. Emergencies do not wait for a review. When a chilled-water pipe starts leaking at 03:00 or a UPS drops to bypass in the middle of a storm, the people on shift must act within minutes, in the right order, without making things worse. They do it with procedures written in advance for exactly those situations: emergency operating procedures, known as EOPs, and the shorter alarm response procedures that sit behind each important alarm. This lesson is about how those procedures are written, kept and used, and why practising them matters as much as having them.

The family of procedures

Jamal has met several kinds of procedure in the program. It helps to see them side by side.

The difference that matters most is time. A MOP is written for a job whose timing Stonebridge chooses. An EOP is written for an event whose timing nobody chooses, and it must work for whoever is on shift when it happens, including a technician in his second month on nights. So EOPs are written to be followed under stress, by people who may be tired, interrupted and frightened, and they are practised until the first steps are known by heart.

What an alarm response procedure contains

The monitoring system raises hundreds of different alarms, and most of them do not need a full emergency procedure. Each alarm that matters has a short alarm response procedure attached to it in the monitoring system, so that the person who sees the alarm can open the response with one click. A good ARP fits on one screen.

• What the alarm means, in plain words, and what usually causes it.
• How to confirm it: which readings or local checks prove the alarm is real and not a sensor fault.
• What to do first, and what not to do.
• Who to tell, and how urgently.
• When it becomes an emergency, and which EOP then applies.

For example, Stonebridge's ARP for a CRAH high return temperature alarm in Hall A tells the technician that the alarm usually means the unit has lost cooling or the aisle has lost containment, asks him to check the unit's chilled-water valve position and supply temperature and to look at the aisle doors and blanking, tells him not to reset or restart the unit until the cause is known, says to tell the shift lead at once, and moves to the EOP for loss of cooling if two or more units alarm together or inlet temperatures in the hall pass a set limit.

What an emergency operating procedure contains

An EOP covers a scenario rather than an alarm: loss of utility power, a UPS failure to bypass, loss of a chiller plant, a chilled-water leak, a fire alarm in a data hall, a generator that fails to start, a release of clean agent, a medical emergency. Stonebridge's EOPs share one format, so that a technician who knows one knows where to look in any of them.

The immediate actions deserve special care. They are the steps that must be taken before there is time to read, so they are kept short, usually three to six steps, and written as plain commands. For the chilled-water leak EOP in Hall A, for example, the immediate actions are: make sure nobody is standing in water near electrical equipment; tell the shift lead and the monitoring desk; find the source if it can be seen safely; and close the isolation valves for that section as shown on the valve chart, but only if the chart shows the other section can carry the load. Everything after that can be read from the page.

Writing procedures people can follow

Stonebridge's EOPs are written by its operations engineers with the shift leads and reviewed with the technicians who will use them. A few rules come from long experience in the industry.

• One action per step, starting with a verb: close valve CHW-A-14, not the valve should be closed if conditions require.
• Name equipment exactly as it is labelled in the field, and include the location: switchboard SWBD-A2, electrical room A-E2.
• Put checks where mistakes are likely: confirm the valve tag before closing it; confirm the partner unit is running before stopping this one.
• Say what success looks like after each important step, so the user knows whether to go on.
• Say when to stop and call for help, and make stopping acceptable: an EOP that has stopped matching what you see is no longer the right EOP.
• Keep it current: an EOP is reviewed when equipment changes, after every use and drill, and at least yearly.

Lesson 1 quoted the industry finding that most human-error outages come from procedures not followed or badly written. EOPs are where that finding bites hardest. A procedure that names a valve that has been replaced, or a phone number that no longer works, or a step that cannot be done in the dark, fails exactly when it is needed. Every review asks the same questions: can this be done as written, by the least experienced qualified person on shift, at night, under pressure?

Where procedures live

The EOPs are kept in three places. The master copies are in the document system, with revision control so that only the current version is in use. Printed copies of the current versions sit in a red binder at the shift lead's desk in the operations centre, because the document system may be on a network that is affected by the emergency. And at key equipment, such as each electrical room, the chilled-water valve stations and the generator yard, laminated placards show the immediate actions and the local valve or breaker chart. When an EOP is revised, the shift lead replaces the binder pages and the placards and signs the change log, and the old versions are removed and destroyed.

Drills

A procedure that has never been used is a guess. Stonebridge practises its EOPs in three ways. Tabletop exercises gather a shift crew around a table to talk through a scenario step by step, with the shift lead playing the events: the alarm appears, the reading is this, the vendor does not answer. Walk-through drills take the crew to the equipment and have them point to each valve, breaker and reading the EOP names, without operating anything. And for some EOPs, live tests are done under a method of procedure, such as the annual pull-the-plug test of DCT-220, which exercises the utility loss EOP for real. Every crew does at least one drill a month, rotating through the EOPs so that each crew practises each important EOP at least once a year.

Drills are timed and recorded. The point is not to grade the crew but to find out whether the procedure and the people work together. A step that takes much longer than expected, a valve that is hard to reach, a contact who does not answer or a step that two people understand differently are all findings, and each one becomes a change to the procedure, the equipment or the training.

# EOP step tracker: record each step's time and check order and duration against the procedure
eop = [  # step, expected maximum minutes from start
    ("people safe, area clear of water near electrical equipment", 2),
    ("shift lead and monitoring desk told", 3),
    ("source found and confirmed", 8),
    ("section isolation valves closed per valve chart", 12),
    ("other section confirmed carrying the load", 15),
    ("tenants and site manager notified", 20),
]
log = [  # step index, minutes from start (from the drill record)
    (0, 1.5), (1, 2.5), (2, 9.5), (3, 13.0), (4, 14.5), (5, 26.0),
]
last = -1
for idx, t in log:
    step, limit = eop[idx]
    notes = []
    if idx != last + 1:
        notes.append("out of order")
    if t > limit:
        notes.append(f"{t - limit:.1f} min over")
    last = idx
    print(f"{t:5.1f} min  {step:60} {'; '.join(notes) if notes else 'ok'}")

  1.5 min  people safe, area clear of water near electrical equipment   ok
  2.5 min  shift lead and monitoring desk told                          ok
  9.5 min  source found and confirmed                                   1.5 min over
 13.0 min  section isolation valves closed per valve chart              1.0 min over
 14.5 min  other section confirmed carrying the load                    ok
 26.0 min  tenants and site manager notified                            6.0 min over

In this drill, finding the leak took longer than the procedure allowed, because the leak was simulated under the raised floor at the far end of the row and the crew had to lift tiles to find it, which also made the valves close a minute late; and the notification step ran six minutes over because the tenant contact numbers were in a different binder. Both findings went into the drill record. The first led to a request for leak detection cable zoning to be shown on the floor map so the location of an alarm is clearer; the second to the contacts being added to the EOP itself.

Using an EOP for real

When an emergency happens, the shift lead decides which EOP applies, says so aloud, and assigns roles: who reads the procedure, who carries out the steps, who watches the rest of the plant, who handles calls. One person reads each step aloud and the person doing it repeats it back before acting, then reports the result. This read-and-repeat method, borrowed from aviation and power plants, catches mistakes like the wrong valve tag before they become actions. The times of key steps are written down as they happen, on the EOP's own record sheet or in the shift log, because they are needed later for the incident report and the root cause analysis of Lessons 7 and 9.

Two situations need judgement. If the event does not match any EOP exactly, the shift lead chooses the closest one and uses its safety and stabilization steps while calling for help. And if, partway through, what the crew sees stops matching what the procedure expects, they stop and reassess rather than pressing on. Following a procedure is not the same as following it off a cliff. In both cases the incident management of Lesson 7 takes over as soon as there is someone to run it.