Change management: risk, approval and change windows

Data centres are most at risk not when equipment fails on its own, but when people are working on it. Opening the wrong breaker, closing a valve on the wrong loop, updating firmware on both UPS systems at once, or forgetting to take a CRAH out of maintenance mode: each of these has caused real outages somewhere. Change management is the discipline of planning, approving, carrying out and recording every change to a live facility so that the risk is understood and controlled before anyone touches anything. This lesson covers how changes are classified, assessed and approved, and how change windows and freezes protect customers. Lesson 10 covers the procedure documents that the work itself follows.

Why change management matters

Uptime Institute's outage research consistently finds that human error contributes to a large share of significant outages, roughly two-thirds to four-fifths by its estimates, and that the most common forms are staff not following procedures and procedures that were themselves faulty. In its 2025 report, among organizations that had a major outage involving human error, failure to follow procedures was cited in a clear majority of cases. These are not careless people; they are skilled people working on complex systems without enough structure. Change management is that structure.

The ITIL framework calls this practice change enablement, a name kept in ITIL Version 5, released in 2026, and taught in the ITIL courses (ITIL-141). Data centres apply the same principles to physical infrastructure, with more emphasis on safety and on the physical sequence of steps.

What counts as a change

At Stonebridge a change is any planned addition, removal or modification that could affect the facility's power, cooling, fire protection, security, monitoring or network infrastructure, or a customer's service. That includes obvious work, such as replacing a UPS module or adding a busway tap-off, and less obvious work: updating BMS software, changing an alarm limit (lesson 3), moving a sensor, altering a CRAH's setpoint, adding a rack to a row, or changing a door's access schedule. Routine operation within normal limits, such as a technician reading a meter, and customer work inside their own equipment, are not facility changes, though customer work that affects the facility, such as adding servers that raise a rack's power, is captured through requests and capacity checks.

Types of change

Changes are sorted by how much review they need:

• Standard changes: low-risk, frequent, well-understood changes with a pre-approved procedure, such as installing a tap-off box using the standard MOP, replacing a CRAH filter, adding a server to a customer rack, or making an alarm limit change already approved by rationalization. They are recorded but need no individual approval.
• Normal changes: everything else that is planned. They are assessed for risk and approved at the level the risk requires, from the shift lead to the change advisory board.
• Emergency changes: changes needed urgently to restore service or prevent imminent harm, such as isolating a leaking pipe section or replacing a failed controller to get a CDU back. They are approved quickly by a small group, often the on-call engineer and the critical facilities manager, and reviewed afterwards.

A healthy change process has many standard changes, a steady number of normal ones and few emergencies. A rise in emergency changes usually means that problems are not being found early enough, or that people are using the emergency route to avoid review.

Assessing risk

Each normal change is assessed by asking what could go wrong and how bad it would be. Stonebridge uses a simple matrix: the impact if the change goes wrong (from 1, no effect on customers, to 5, loss of customer load across a hall) multiplied by the likelihood (from 1, very unlikely, to 5, likely), giving a score from 1 to 25. The score sets the approval level and the controls required.

Impact is judged from the state the facility will be in during the work. Work on one side of a 2N system leaves the load on the other side alone: the impact of a second failure during the work is high, even if the work itself is simple. Work that reduces redundancy (taking a UPS module, CRAH or generator out) raises impact; work done with full redundancy intact lowers it. Likelihood is judged from the complexity of the work, how familiar the team is with it, whether a tested procedure exists and whether the work can be undone quickly.

# Change risk scoring and approval level
def approval(impact, likelihood):
    score = impact * likelihood
    if score <= 4:  level = "shift lead"
    elif score <= 9: level = "critical facilities manager"
    elif score <= 15: level = "change advisory board"
    else: level = "CAB + site manager + affected customers' agreement"
    return score, level

changes = [  # (change, impact, likelihood)
    ("Replace CRAH-A03 filters (standard MOP)", 1, 1),
    ("Add Hall D flow-deviation alarm in BMS", 2, 2),
    ("Replace branch breaker 2-17 on PDU-B2-05", 3, 2),
    ("UPS-B2 module firmware update", 4, 3),
    ("Swap Hall D busway BW-D1-04 feed breaker", 5, 4),
]
for name, i, l in changes:
    s, lvl = approval(i, l)
    print(f"{name:45s} impact {i} x likelihood {l} = {s:2d} -> {lvl}")

Replace CRAH-A03 filters (standard MOP)       impact 1 x likelihood 1 =  1 -> shift lead
Add Hall D flow-deviation alarm in BMS        impact 2 x likelihood 2 =  4 -> shift lead
Replace branch breaker 2-17 on PDU-B2-05      impact 3 x likelihood 2 =  6 -> critical facilities manager
UPS-B2 module firmware update                 impact 4 x likelihood 3 = 12 -> change advisory board
Swap Hall D busway BW-D1-04 feed breaker      impact 5 x likelihood 4 = 20 -> CAB + site manager + affected customers' agreement

The numbers are a guide, not a substitute for thought. The value of the matrix is that it makes the people planning the change say out loud what could go wrong and how bad it would be, and it gives everyone the same scale.

The change request

A normal change starts as a change request ticket. Stonebridge's form asks for: what is being changed and why; the systems and customers affected; the redundancy state during the work; the risk score and its reasoning; the date, time and expected duration; who will do the work and who will supervise; the method of procedure (lesson 10) or a link to it; the back-out plan if it goes wrong; how success will be checked; and what records will be updated afterwards. A change request that cannot answer these questions is not ready to approve.

The change advisory board

Higher-risk changes are reviewed by the change advisory board (CAB), a weekly meeting of the critical facilities manager, the electrical and mechanical engineers, the operations shift leads, a security representative and, when their space is affected, the account managers who speak for customers. The CAB does not do the technical design of a change. It checks that the change has been thought through, that its timing does not clash with other work or with customer events, that the redundancy state during the work is acceptable, and that customers have been told. It can approve, reject, or send a change back with questions.

Technicians attend the CAB when their changes are reviewed, and often notice practical problems the engineers miss: the generator test scheduled for the same night, the delivery that will block the corridor, the CRAH that is already out of service in the same hall.

Change windows and freezes

A change window is a period agreed in advance, often in customer contracts, when changes that carry risk to customer services may be made. Stonebridge's standard window for normal changes in shared halls is Tuesday to Thursday, 22:00 to 05:00, with the customers' agreement for anything that reduces redundancy on their power or cooling. Low-risk standard changes can happen any time. Some customers have their own windows: Corvid AI asks that Hall D changes avoid the first week of each month, when its largest training jobs start.

A change freeze is a period when no normal changes are made except emergencies: around major holidays, during a customer's critical business period such as Bayview's year-end processing, or after a serious incident until the cause is understood. Freezes reduce risk at the times when an outage would cost most, and they are posted on the change calendar.

# Change calendar check: conflicts between proposed changes and existing plans
from datetime import datetime as D
calendar = [  # (start, end, item, systems)
    (D(2027, 3, 16, 22), D(2027, 3, 17, 2), "GEN-A block monthly load test", {"GEN-A", "SWBD-A2", "Hall A"}),
    (D(2027, 3, 17, 22), D(2027, 3, 18, 4), "UPS-B2 module firmware update", {"UPS-B2"}),
    (D(2027, 3, 29, 0), D(2027, 4, 2, 0), "Bayview year-end freeze", {"Hall A"}),
]
proposed = [
    (D(2027, 3, 16, 23), D(2027, 3, 17, 1), "Replace breaker 2-17 on PDU-B2-05", {"PDU-B2-05", "UPS-B2", "Hall A"}),
    (D(2027, 3, 17, 23), D(2027, 3, 18, 1), "CRAH-A07 actuator replacement", {"CRAH-A07", "Hall A"}),
    (D(2027, 3, 30, 22), D(2027, 3, 31, 1), "Add tap-off box BW-D1-04", {"BW-D1-04", "Hall D"}),
]
def overlap(a1, a2, b1, b2): return a1 < b2 and b1 < a2
for s, e, name, systems in proposed:
    clashes = [item for cs, ce, item, csys in calendar if overlap(s, e, cs, ce)]
    shared = [item for cs, ce, item, csys in calendar if overlap(s, e, cs, ce) and (systems & csys or "Hall A" in csys and "Hall A" in systems)]
    print(f"{name:36s}: {'conflict with ' + ', '.join(shared) if shared else ('same time as ' + ', '.join(clashes) + ' (no shared systems)' if clashes else 'clear')}")

Replace breaker 2-17 on PDU-B2-05   : conflict with GEN-A block monthly load test
CRAH-A07 actuator replacement       : same time as UPS-B2 module firmware update (no shared systems)
Add tap-off box BW-D1-04            : same time as Bayview year-end freeze (no shared systems)

The first proposed change would replace a B-side branch breaker while the A-side generator block is being load tested, so for those two hours a rack could be relying on the A side while its generators are under test. The CAB moved it to the next night. The actuator change overlaps the UPS-B2 firmware work in time but shares no systems except the hall, and was allowed. The tap-off box falls during Bayview's freeze but is in Hall D, outside the frozen hall, so it may go ahead.

Telling customers

Customers must know about changes that could affect them. Stonebridge sends a maintenance notice through the portal, usually at least ten business days before a normal change that reduces redundancy on a customer's power or cooling, and at least 48 hours before any other customer-affecting change. The notice says what will be done, when, what the customer might notice (for example, one feed to their racks off for 30 minutes), what they should check beforehand (that every device is dual-corded and both feeds are live), and who to contact. Emergency changes are notified as soon as possible, often by phone first.

Customers sometimes object or ask for a different time. Their concerns are taken to the CAB, and a change that a customer has good reason to oppose is usually moved. The point of the notice is not to tell customers what will happen to them but to let them prepare and, where necessary, to protect themselves.

After the change

Every change ends with a closure step: was it completed as planned, partly, or backed out? Were there any unexpected events? Were the records updated (DCIM, drawings, labels, alarm database)? A change that caused an incident, or was backed out, gets a short review to learn why. Monthly reports track the number of changes by type, the share completed successfully first time, and the number that caused incidents. Stonebridge's target is that fewer than one change in 200 causes a customer-affecting incident.