Operating a critical facility around the clock

Jamal Pierce begins the last technical term of his program at Stonebridge Data with a new kind of assignment. Until now his work has been organized around systems: racks, cabling, power, cooling, servers and networks. This term is organized around time. A data centre never closes, and the people who keep it running work in shifts that cover every hour of every day of the year, including the night of a storm, the morning of a holiday and the afternoon when three tenants want work done at once. Roz Kerr, his mentor, puts it simply on the first morning: building a data centre is a project that ends; operating one is a job that never does. This course is about that job: keeping equipment healthy through maintenance, responding well when something goes wrong, finding out why it went wrong, and passing the facility safely from one shift to the next.

The course follows Jamal through a season on the YUL-2 campus in Beauharnois. He joins Dmitri Volkov's crew on the operations rotation, works with the maintenance planner and the vendors who service the plant, takes part in an incident from first alarm to root cause, and learns to write the logs and hand-offs that hold the operation together. It builds on everything earlier in the program: the safety rules of DCT-120, the power chain of DCT-220, the cooling plant of DCT-230 and the monitoring, tickets, change control and methods of procedure of DCT-330. Where DCE-450 in the engineering school looks at reliability and operations from the designer's and the manager's desk, this course stays on the floor, with the technician who carries the radio.

Why operations decides reliability

A data centre's design sets how reliable it can be. Its operation decides how reliable it is. Two buildings with the same redundant power and cooling can have very different records, because one is maintained on schedule by people who follow good procedures and the other is not. Industry surveys make the point year after year. Uptime Institute's 2025 annual outage analysis reported that outages had become less frequent and less severe for the fourth year in a row, but that nearly 40 percent of organizations had suffered a major outage caused by human error in the past three years. Of those human-error outages, about 85 percent came from staff not following procedures or from flaws in the procedures themselves, and failures to follow procedures had risen compared with the year before. Power problems remained the most common cause of serious outages.

Those numbers describe the work of this course. A redundant system protects the load only if its redundancy is actually available when something fails, which means maintenance has been done, nothing has been left in the wrong position and the people on shift know the state of the plant. A well-written procedure protects the load only if it is followed, step by step, by someone who understands why each step is there. And an incident teaches something only if someone takes the time to find its real cause and fix it. The equipment does not do these things by itself.

The operations team

Stonebridge runs each campus with a small permanent operations team and a wider circle of specialists. The team Jamal joins is typical of a colocation operator of this size.

The operations team does not operate the tenants' IT equipment, but it is responsible for everything that keeps that equipment running: power to the cabinet, cooling, fire protection, security and the building itself. Tenants feel the results of operations directly, so they watch it closely. Corvid AI's contract, for example, includes a maintenance notification clause, an incident notification clause and a right to see the reports from any incident that affects Hall D.

Shifts and coverage

The campus is staffed twenty-four hours a day by four crews working twelve-hour shifts, days from 07:00 to 19:00 and nights from 19:00 to 07:00. Each crew works a repeating pattern of two days on, two off, three on, two off, two on and three off, so that over two weeks every crew works seven shifts and every shift is covered. A crew is a shift lead and at least two critical facilities technicians on site at all times, plus the monitoring desk. On weekdays the day shift is joined by the site manager, the planner and the specialist staff, so planned maintenance is usually scheduled then. The night shift is smaller, but it is never less than the minimum: if a technician calls in sick, someone stays over or comes in.

Why does minimum staffing matter so much when the monitoring system can raise alarms and most equipment runs itself? Because the first minutes of a problem are usually decided by people on site. A chilled-water leak, a tripped breaker or a generator that will not start needs someone who knows the plant to look, think and act, often before a specialist can be reached. Uptime Institute's guidance on management and operations treats qualified, round-the-clock site staffing as one of the foundations of reliable operation, alongside training, maintenance and procedures. A campus that saves money by leaving the night shift short is borrowing against its next incident.

# Shift coverage check for a two-week rotation (D = day, N = night, - = off)
rota = {
    "A": "DD--NNN--DD---",
    "B": "--DD---NN--DDD",
    "C": "NN--DDD--NN---",
    "D": "--NN---DD--NNN",
}
days = len(next(iter(rota.values())))
problems = 0
for d in range(days):
    day = [c for c, r in rota.items() if r[d] == "D"]
    night = [c for c, r in rota.items() if r[d] == "N"]
    if len(day) != 1 or len(night) != 1:
        problems += 1
        print(f"day {d + 1}: day shift {day}, night shift {night}")
shifts = {c: sum(ch != "-" for ch in r) for c, r in rota.items()}
print("shifts per crew in two weeks:", shifts)
print("coverage gaps or overlaps:", problems)

shifts per crew in two weeks: {'A': 7, 'B': 7, 'C': 7, 'D': 7}
coverage gaps or overlaps: 0

The check confirms what the pattern was built to do: every day and every night has exactly one crew, and each crew works seven shifts in fourteen days. Dmitri keeps a copy of a check like this with the rota, because swaps between technicians are common, and a swap entered on the wrong day can quietly leave a night with nobody qualified in the electrical rooms.

A shift in outline

Every shift has the same skeleton, whatever else happens in it.

• Hand-off: the outgoing and incoming crews go through the open items, the redundancy status and anything unusual, as Lesson 11 describes. The incoming lead signs for the facility.
• Rounds: technicians walk the plant on a fixed route, reading gauges and panels, looking, listening and smelling, and recording what they find.
• Planned work: maintenance, vendor visits and tenant work under tickets and methods of procedure, as DCT-330 describes.
• Responsive work: alarms, tickets and customer requests that arrive during the shift.
• Records: the shift log, the round sheets, completed work orders and anything that must be passed on.

Rounds deserve more attention than new technicians often give them. The monitoring system sees what it has sensors for. A technician on a round sees a drip forming under a valve, hears a fan bearing beginning to growl, smells the warm insulation of an overloaded cable, notices a door propped open or a fire extinguisher missing. Roz's first rule for rounds is that a round is not a walk: you stop at each point, read the value, compare it with the last reading and the normal range, and write it down. A reading that has moved since yesterday matters even if it is still inside its limits.

The three things operations protects

Roz tells every new technician that operations protects three things, in this order: people, the critical load and the equipment. People come first without exception: nobody is asked to take a risk to keep a server running. The critical load comes next: the power, cooling and connectivity that tenants' equipment depends on. The equipment itself comes last, because equipment can be repaired or replaced, and protecting it at the cost of the load or of a person is the wrong trade. When decisions have to be made quickly, as they do in an incident, this order settles most arguments before they start.

Protecting the load has a particular meaning in a redundant facility. Most failures do not drop the load, because a second path carries it. What they do is remove redundancy: after a UPS module fails in Hall A, the load is still on protected power, but the next failure could drop it. Operations treats the loss of redundancy as a problem in its own right, to be reported, recorded on the shift lead's redundancy whiteboard and fixed, not as a non-event because nothing went down. Lesson 4 returns to this when it looks at maintenance, which also removes redundancy, deliberately.

Reliability numbers a technician should know

Operations speaks a little in numbers, and three come up constantly. Availability is the fraction of time a system is able to do its job. Mean time between failures, MTBF, is the average operating time between failures of a repairable item. Mean time to repair, MTTR, is the average time from a failure to the item working again, including the time to notice, reach, diagnose and fix it. The three are linked: availability is MTBF divided by the sum of MTBF and MTTR.

The formula shows why operations matters even for very reliable equipment. A component that fails on average once every five years is very good, but if each failure takes three days to repair because the spare must be shipped and the vendor flown in, the component's availability is lower than one that fails twice as often but is fixed in four hours. Much of operations is shortening MTTR: spares on site, vendors on contract with response times, procedures written in advance and technicians who know the plant.

# Availability from MTBF and MTTR, and what shortening repair time is worth
hours_per_year = 8760
cases = [
    ("rare failure, slow repair", 5 * hours_per_year, 72),
    ("twice as often, fast repair", 2.5 * hours_per_year, 4),
    ("twice as often, fast repair, spare on site", 2.5 * hours_per_year, 1.5),
]
for name, mtbf, mttr in cases:
    a = mtbf / (mtbf + mttr)
    down = (1 - a) * hours_per_year * 60
    print(f"{name:44} availability {a * 100:.4f} %  ~{down:5.0f} min down per year")

rare failure, slow repair                    availability 99.8359 %  ~  863 min down per year
twice as often, fast repair                  availability 99.9817 %  ~   96 min down per year
twice as often, fast repair, spare on site   availability 99.9932 %  ~   36 min down per year

The first case loses about fourteen hours a year on average; the third about thirty-six minutes, although its equipment fails twice as often. These numbers are averages for one component, and a redundant system is far better than any single component in it. But the lesson holds at every scale: the speed and quality of the response is part of the reliability of the facility.

Working in the operation

Joining the operation changes what is expected of Jamal. On the installation crews of earlier courses he worked one job at a time under a ticket. On shift he is part of a crew responsible for the whole campus for twelve hours, and he must keep a picture of the whole in his head: what is in maintenance, what is in alarm, which vendors are on site, which tenant has work under way and what the weather is doing. Dmitri expects three habits from every technician on his crew. Report everything unusual, even if it seems minor; the shift lead decides what matters. Never change the state of anything without the shift lead knowing, whether that is opening a valve, resetting an alarm or closing a door that was propped open for a reason. And write it down: if it is not in the log, the next shift does not know it happened.

The rest of the course builds on this picture. Lessons 2 to 5 look at maintenance: the strategies, the program, doing it safely on live systems and the condition monitoring that finds problems early. Lessons 6 to 10 look at what happens when something goes wrong: emergency procedures, incident response, communication, root cause analysis and the follow-up that stops it happening again. Lesson 11 covers shift hand-offs and documentation, and Lesson 12 follows a real incident in Hall A from the first alarm to the last corrective action.