Methods of procedure and standard operating procedures

Change management decides whether a piece of work should happen and when. The procedure document decides how. In data centers, the two main kinds are the method of procedure (MOP), a step-by-step script for a specific piece of work on live infrastructure, and the standard operating procedure (SOP), a reusable instruction for a routine task. A third kind, the emergency operating procedure (EOP), tells staff what to do when something has already gone wrong; DCT-340 covers those. This lesson explains what goes into a MOP and an SOP, how they are written, reviewed and used, and why the best technicians follow them even when they know the work by heart.

MOP, SOP and EOP

Many standard changes (lesson 9) are carried out with a standard MOP: a MOP written once for a repeated task, such as installing a tap-off box, and reused with the specific equipment names filled in each time. Normal changes usually need a MOP written or adapted for that change.

What goes into a MOP

Stonebridge's MOP template has the same sections every time, so that anyone can find what they need:

• Header: MOP number and revision, the change ticket, title, site, systems affected, author, reviewers and approvals with dates.
• Scope and purpose: what will be done and why, in two or three sentences.
• Risk summary: the redundancy state during the work, the customers affected, the risk score from the change request, and the specific hazards to people, such as arc flash boundaries or pressurized coolant.
• Prerequisites: everything that must be true before step 1: change approved, customer notices sent, parts and tools on site and checked, people present, redundancy board green except for the planned item, no conflicting work, weather acceptable for outdoor work.
• Roles: who does each step, who reads and checks, who is the point of contact with the desk, and who can stop the work.
• Procedure steps: numbered, one action each, with the expected result and a verification for each step.
• Hold points: steps where work stops until a named person confirms it may continue.
• Back-out plan: steps to return to the starting state, and the point after which backing out is no longer possible.
• Post-work checks and close-out: confirming normal state, clearing maintenance modes and suppressions, updating DCIM, drawings and labels, notifying customers and closing the change.

The template is long on purpose. A MOP that fits on one page is usually missing its prerequisites or its back-out plan, which are the sections that prevent the worst outcomes.

Writing good steps

Each step should be a single action that one person does, written so that it can only be read one way. It names the exact equipment by its label, says what to do and what should happen, and gives a way to check that it did. Compare:

Every step that changes the state of equipment has a verification by a second source, usually the desk watching the BMS or EPMS. This is the reader-doer practice used in aviation and power plants: one person reads the step aloud, the other does it and reports the result, and the reader confirms against the expected result before moving on. It sounds slow; it takes seconds per step and catches the moment someone reaches for breaker 2-18 instead of 2-17.

Back-out plans and points of no return

Every MOP must say how to undo the work if something goes wrong, and when undoing is no longer possible. For a breaker replacement, the point of no return comes when the old breaker has been removed: from then on, the way back is to finish installing the new one. The MOP states this, and the prerequisites make sure the risky part starts only when everything is ready: the new breaker checked against the drawing, the right tools present and the customer confirmed dual-corded.

The back-out plan also says who decides to back out and on what signs: an unexpected alarm, a reading different from the expected result, a customer report of lost devices, or simply the work running much longer than planned and approaching the end of the change window.

Reviewing and approving a MOP

A MOP is reviewed by at least one person who did not write it, usually an engineer who owns the system and a senior technician who will do or supervise the work. Reviewers check the steps against the drawings (lesson 11), the equipment labels and the actual state of the facility. Stonebridge requires a site walk-down for normal changes: before approval, the people doing the work walk through the steps at the equipment, reading each label and checking each breaker, valve and point named in the MOP. Walk-downs routinely find mistakes: a breaker relabeled since the drawing was made, a valve that is not where the drawing puts it, a step that needs a ladder.

Once approved, the MOP's revision is fixed. If anything has to change on the night, the work stops at a safe point, the change is agreed with the approver by phone, written on the MOP with the time and names, and the MOP is revised formally afterwards. Changing a step on the fly without stopping is how most procedure-related incidents start.

# MOP checker: every step that changes equipment state needs a verification and
# a rollback before the point of no return; hold points need a named approver.
mop = [  # (step, action, changes state?, verification, rollback, hold approver)
    (1, "Confirm prerequisites list complete", False, "shift lead initials", None, None),
    (2, "Set PDU-B2-05 branch 2-17 alarms to maintenance mode (CHG-5512)", True, "desk sees mode on BMS", "clear maintenance mode", None),
    (3, "Customer confirms all devices dual-corded and A side healthy", False, "customer by phone", None, "shift lead"),
    (4, "Open breaker 2-17", True, "EPMS shows open, A-side current about 22 A", "close breaker 2-17", None),
    (5, "Remove old breaker (point of no return)", True, "", None, "shift lead"),
    (6, "Install and torque new breaker", True, "torque recorded", None, None),
    (7, "Close breaker 2-17", True, "EPMS shows closed, both sides about 11 A", None, None),
    (8, "Clear maintenance mode", True, "", None, None),
]
no_return = 5
problems = []
for step, action, changes, verify, rollback, hold in mop:
    if changes and not verify:
        problems.append(f"step {step}: changes equipment but has no verification")
    if changes and step < no_return and not rollback:
        problems.append(f"step {step}: before the point of no return but has no rollback")
    if "point of no return" in action and not hold:
        problems.append(f"step {step}: point of no return needs a hold point")
print("\n".join(problems) if problems else "MOP structure OK")

step 5: changes equipment but has no verification
step 8: changes equipment but has no verification

The checker finds two gaps: the point-of-no-return step has no verification (what should the electrician confirm after removing the old breaker, for example that the bus stab is undamaged?), and clearing maintenance mode has no verification (the desk should confirm that the alarms are live again). Both were added before approval. The second gap is exactly the kind of omission the capstone shows going wrong.

Using a MOP on the night

On the night, the MOP is printed or on a tablet, and the person leading the work keeps the controlled copy. Each step is read, done, verified and initialed with the time. Nobody skips ahead, even for steps that seem trivial. The desk keeps its own copy and follows along, because it does many of the verifications and because it needs to know what state the facility is in at each moment. If the radio goes quiet for longer than the step should take, the desk calls.

Two habits matter most. The first is stopping when reality does not match the MOP: a label different from the step, a reading different from the expected result, an alarm the MOP did not predict. Stopping costs minutes; carrying on with a wrong picture can cost the load. Anyone in the team may call a stop, including the most junior technician, and nobody is criticized for a stop that turns out to be unnecessary. The second habit is completing the close-out with the same care as the work: the facility is not back to normal until maintenance modes are cleared, suppressions removed, alarms live, records updated and the desk has confirmed the redundancy board.

Standard operating procedures

SOPs cover the routine work that keeps the facility running: walking rounds, shift handovers, visitor escorts, receiving deliveries, generator weekly checks, filter changes, battery visual inspections. They are shorter than MOPs and do not need individual approval each time they are used, but they are still controlled documents with owners, revision numbers and review dates. Stonebridge reviews every SOP at least once a year, and after any incident where it was involved.

SOPs make work consistent between people and shifts, and they are how new staff learn. Jamal's first weeks on the desk were largely spent working through the desk SOPs with Dmitri: alarm response, shift log, handover, ticket handling, on-call paging and the flood procedure.

Document control

Procedures are only useful if everyone uses the current version. Stonebridge keeps all MOPs, SOPs and EOPs in a document management system where each has a number, a revision, an owner and an approval record; superseded revisions are withdrawn and marked, and printed copies are dated and marked 'valid for this change only' or 'check revision before use'. A technician who finds a procedure that is wrong or out of date raises a ticket to its owner, rather than correcting their own copy. A procedure fixed quietly in one person's folder is still wrong for everyone else.