Operations: staffing, procedures and change control

Once commissioning is complete and the turnover accepted, the facility belongs to the operations team for the next twenty years or more. Studies of data centre outages consistently find that human and management factors, rather than equipment, are behind most significant incidents: procedures not followed, procedures that were wrong, staff not trained for the situation, or changes made without understanding their effects. This lesson covers how a data centre operations team is organized and staffed, the procedures it works from, how it is trained and drilled, how changes to the facility are controlled, and the standards and audits that assess operational maturity.

The operations organization

A typical data centre operations organization has a site or facility manager responsible for the infrastructure; a critical facilities team of shift leads and electrical, mechanical and controls technicians who run the plant around the clock; a maintenance planner who schedules preventive work and coordinates vendors; and support functions such as health and safety, security, and an engineering role that handles analysis, capacity planning and changes. Many tasks are performed by vendors under service contracts, especially UPS, generator, chiller and fire system maintenance, but the operations team remains responsible for controlling and supervising that work.

Organizations choose between in-house staff, outsourcing to a facilities management company, or a mix. Whatever the model, the owner needs enough in-house expertise to understand the plant, set standards and hold contractors to them. A common failure is to outsource operations completely and lose the ability to judge whether the work is being done well.

Staffing around the clock

Critical facilities are staffed 24 hours a day, 365 days a year, because the time to respond to an alarm, often minutes, determines whether a single failure becomes an outage. Covering one post continuously takes far more than three people working eight-hour shifts, because each person works only part of the year's hours once vacations, holidays, sickness and training are taken out.

# How many people does one 24/7 operations post need?
hours_to_cover = 24 * 365
paid_hours = 40 * 52
absences = {"vacation": 15 * 8, "paid holidays": 10 * 8, "sick and personal": 6 * 8,
            "training and drills": 80}
productive = paid_hours - sum(absences.values())
fte_per_post = hours_to_cover / productive
print(f"Paid hours per person: {paid_hours}; productive on shift: {productive}")
for name, h in absences.items():
    print(f"  less {name:<20} {h:4d} h")
print(f"People needed per 24/7 post: {fte_per_post:.2f}")

posts = {"Shift lead": 1, "Electrical technician": 1, "Mechanical technician": 1, "Controls and monitoring": 1}
total = sum(posts.values()) * fte_per_post
print(f"Minimum crew of {sum(posts.values())} on every shift needs {total:.1f} people")
print(f"With a 12 % allowance for turnover and overlap at handover: {total * 1.12:.1f}")

Paid hours per person: 2080; productive on shift: 1752
  less vacation              120 h
  less paid holidays          80 h
  less sick and personal      48 h
  less training and drills    80 h
People needed per 24/7 post: 5.00
Minimum crew of 4 on every shift needs 20.0 people
With a 12 % allowance for turnover and overlap at handover: 22.4

Each person provides about 1,750 productive hours a year against 8,760 hours to cover, so each 24/7 post needs about five people. A minimum crew of a shift lead and three technicians on every shift therefore requires about 20 people, and more in practice to allow for turnover, overlap at shift handover and day-shift tasks such as planned maintenance. These numbers explain why staffing is one of the largest operating costs of a data centre, and why there is pressure to reduce shift crews. Reductions must be weighed against the response needed: how many people does it take to carry out the emergency procedures for the worst credible event, simultaneously, before relief arrives?

Shift patterns vary: rotating eight-hour shifts, or twelve-hour shifts with four or five crews. Twelve-hour shifts reduce the number of handovers but can increase fatigue, which matters because fatigue degrades exactly the judgment needed in an emergency. Whatever the pattern, fatigue management rules limit consecutive shifts and overtime.

In Canada, hours of work, overtime and rest periods are set by provincial employment standards legislation (or the Canada Labour Code for federally regulated employers), and occupational health and safety acts require workplace health and safety committees or representatives, hazard assessments and training.

Shift handover

Information is lost at every handover. A structured handover, written and spoken, covers the current plant configuration (what is out of service, on bypass or in manual), active alarms and their status, work in progress and permits open, planned work for the next shift, vendor activity, and anything unusual. Many sites use a handover log and require the incoming lead to walk the critical spaces before accepting the shift. A plant left in an abnormal configuration by one shift and not noticed by the next is a classic precursor to an outage.

Procedures

Data centre operations run on written procedures, organized in a few families:

• Standard operating procedures (SOPs): routine tasks such as rounds, readings, starting and stopping equipment, and normal switching.
• Methods of procedure (MOPs): step-by-step instructions for maintenance and other planned work on critical systems (Lesson 8).
• Emergency operating procedures (EOPs): actions for responding to failures, such as loss of utility, a generator failing to start, a UPS alarm, loss of a chiller or a water leak.
• Alarm response procedures (ARPs): what to do when each significant alarm appears, including how to verify it and whom to call.
• Escalation and communication procedures: who is notified, how, and when, including customers and senior management.

Procedures must match the plant as built, which means they are written or validated during commissioning, and kept current through change control. They should be written for the people who use them: clear steps, one action per step, expected results stated, warnings placed before the step they apply to, and diagrams of the relevant part of the single-line or schematic. A procedure library that is out of date is worse than useless, because it gives false confidence.

Training and drills

New staff need site-specific training even when they are experienced, because every data centre is different. A training programme typically combines classroom sessions on the site's systems and single-line diagrams, walk-downs of the plant, supervised performance of tasks, and qualification sign-offs for each task a person may perform alone. Records show who is qualified for what, and the shift schedule ensures that a qualified person for each critical task is always on site.

Emergency procedures are rarely used for real, so they must be practised. Drills range from tabletop exercises, in which the team talks through its response to a scenario, to walk-through drills at the equipment, in which each step is simulated without operating anything, to live exercises such as an annual pull-the-plug test on a facility designed for it. After each drill, the team records what went well and what did not, and updates procedures and training. A drill that reveals a problem has done its job.

Change control

Every change to the facility, whether a new circuit, a modified control sequence, a firmware update, a relay setting change or a temporary bypass, alters how the facility behaves, and many outages follow a change that seemed minor. Change management gives each proposed change a structured review before it happens:

• A change request describes what will change, why, the risk, the rollback plan and the documents that must be updated.
• The change is classified by risk: standard changes with pre-approved procedures, normal changes requiring review, and emergency changes made to restore service, reviewed afterwards.
• A change advisory board, including operations, engineering and often IT and customer representatives, reviews normal changes and schedules them to avoid conflicts with other work and with high-risk periods (for example, a change freeze during peak business seasons).
• After implementation, the change is verified, and drawings, procedures, settings files, the CMMS and the monitoring system are updated.

Configuration management is the companion discipline: maintaining an accurate record of the facility's current configuration, including drawings, settings, firmware versions, control sequences and temporary changes. Temporary changes, such as a jumper in a control panel or a breaker locked open, are tracked in a register and reviewed regularly, because forgotten temporary changes are a frequent cause of failures.

Contractors and permits to work

Much of the work in a data centre is done by contractors who do not know the site. Control measures include site inductions, escorts or supervision, a permit-to-work system that authorizes specific work in specific places at specific times with specific isolations, and an operations team member who verifies the plant configuration before work starts and after it ends. Hot work, such as welding, needs special permits because of fire detection and suppression systems, and work near live equipment needs electrical safety permits (Lesson 2).

Standards and audits of operations

Several frameworks assess operations rather than design. The Uptime Institute's Management and Operations stamp of approval reviews staffing, training, maintenance, planning and coordination, and operating conditions. ISO/IEC 22237 (based on the European EN 50600 series) includes operational requirements, and ISO 41001 covers facility management systems. Customers also ask for evidence through audits such as SOC 2 reports and ISO/IEC 27001 certification, which include physical and environmental controls. Whatever the framework, the common themes are documented procedures that are actually followed, qualified staff, controlled change and continuous improvement.