Operations review: monitoring, change management and DCIM

Most outages in data centers are not caused by equipment failing on its own. Uptime Institute's annual outage reports have found for years that human error plays a part in roughly two-thirds to four-fifths of outages, and that the most common forms are staff not following procedures and procedures that were wrong in the first place. That is why exams give operations so much weight: monitoring, tickets, change management, methods of procedure and DCIM are the tools that stop people from causing outages. This review follows DCT-330 and adds the availability arithmetic that operations questions often include.

Monitoring systems

Three systems watch a facility, and an exam may ask which one you would open for a given question (DCT-330 lessons 1 and 2).

• The building management system (BMS) monitors and controls mechanical and environmental equipment: CRAHs, CDUs, chillers, pumps, leak detection, temperature and humidity sensors.
• The electrical power monitoring system (EPMS) records the power system: meters, breaker positions, UPS and generator status, and fast events such as voltage sags, often with waveform capture.
• DCIM (data center infrastructure management) holds the model of the site: assets, their positions, their power and network connections, and capacity.

Points reach these systems over protocols such as BACnet, Modbus and SNMP, often through gateways. A technician's monitoring skills include spotting stale data (a value that has not changed for hours), a command that does not match its feedback (a valve commanded to 100 percent but reading 8 percent), and a meter that disagrees with the meter upstream of it.

Alarms

An alarm should mean that someone needs to act. Good alarm design gives every alarm a priority, a limit, a deadband (so it does not clear and return as a value hovers near the limit), a delay (so brief spikes do not alarm) and a written response (DCT-330 lesson 3). Stonebridge uses four priorities in lower case on its screens: critical, major, minor and advisory, aiming for a pyramid of roughly 1, 5, 15 and 80 percent of alarms.

Alarm management guidance in ANSI/ISA-18.2 and EEMUA 191 gives targets such as about one alarm per 10 minutes per operator as acceptable and two as the most an operator can manage, with a flood defined as more than ten in 10 minutes. The alarm response procedure is acknowledge, assess, act, escalate and record (DCT-330 lesson 4). During a flood, find the cause first, on the redundancy board and one-line, then deal with critical alarms.

Tickets and escalation

Every piece of work and every incident has a ticket. A ticket records the request, its priority, who owns it, every update and the outcome. Priorities P1 to P4 carry response and resolution targets, and an SLA timer runs from the moment the ticket opens. Escalation is functional (to someone with different skills) or hierarchical (to someone with more authority), and the escalation matrix says who and when (DCT-330 lesson 7). Remote hands requests from customers are verified with two identity checks against the customer's authorized list before any work starts (DCT-330 lesson 8).

Change management

A change is any planned addition, modification or removal that could affect service. Change management classifies changes (standard, normal, emergency), scores their risk, sends them for the right level of approval, and schedules them in change windows that avoid busy periods and freezes (DCT-330 lesson 9). ITIL calls this practice change enablement.

Stonebridge scores risk as impact times likelihood, each from 1 to 5. The key idea, and a favorite exam point, is that impact is judged from the redundancy state during the work, not in normal running. Moving a CDU group from N+1 to N for a sensor change makes any further failure a loss of cooling to the row, so the impact score is high even though the work itself is simple.

Methods of procedure

A method of procedure (MOP) is the step-by-step script for a change. Each step says what to do, who does it, what to check afterwards, and how to back out. Hold points require a check and a go decision before continuing; a point of no return marks the step after which backing out is no longer a simple reversal. Before the work, the team walks down the MOP at the equipment, checking every tag and label against the document (DCT-330 lesson 10). During the work, one person reads each step and another performs it, and the reader confirms each result before moving on. In the DCT-330 capstone, that read-back caught the controls contractor putting the wrong CDU into maintenance mode, and the team stopped, backed out and resumed safely.

Standard operating procedures (SOPs) cover routine tasks; emergency operating procedures (EOPs) cover failures; and all of them are controlled documents with versions and approvals. Using an old printed copy is a classic exam trap and a classic real cause of mistakes.

DCIM and capacity

DCIM lets a technician find any asset, trace its power from rack PDU outlet back to the UPS, trace its network connections, and see the floor map (DCT-330 lesson 5). Its value depends on accuracy, so every install, move and removal must update it, with a second-person check for power moves. Capacity tracking compares what is used and committed with what is available for space, power, cooling and network, after redundancy. Stranded capacity is capacity that exists but cannot be used because another resource has run out first, such as empty rack space in a row with no cooling left (DCT-330 lesson 6).

Availability arithmetic

Availability is the fraction of time a service is up. It is often given as a number of nines: 99.9 percent (three nines) allows about 8.8 hours of downtime a year; 99.99 percent about 53 minutes; 99.999 percent about 5.3 minutes. For components in series, both must work, so availabilities multiply. For redundant components in parallel, the service fails only if all fail, so unavailabilities multiply.

# Availability and change-risk review
HOURS = 8760
def downtime(avail):
    h = (1 - avail) * HOURS
    return f"{h:.2f} h" if h >= 1 else f"{h * 60:.1f} min"
for nines in (0.999, 0.9999, 0.99999):
    print(f"{nines * 100:.3f} % available -> {downtime(nines)} down per year")
path = 0.9995 * 0.9999
pair = 1 - (1 - path) ** 2
print(f"one path {path * 100:.3f} %, 2N pair {pair * 100:.5f} %")

def approval(impact, likelihood):
    score = impact * likelihood
    if score <= 4:  band = "shift lead"
    elif score <= 9: band = "critical facilities manager"
    elif score <= 15: band = "change advisory board"
    else: band = "CAB, site manager and customer notice"
    return score, band
for name, i, l in (("CDU sensor, group N+1", 2, 2), ("CDU sensor, group at N", 5, 2), ("UPS-A2 to bypass", 5, 4)):
    s, b = approval(i, l)
    print(f"{name:<24} score {s:>2} -> {b}")

99.900 % available -> 8.76 h down per year
99.990 % available -> 52.6 min down per year
99.999 % available -> 5.3 min down per year
one path 99.940 %, 2N pair 99.99996 %
CDU sensor, group N+1    score  4 -> shift lead
CDU sensor, group at N   score 10 -> change advisory board
UPS-A2 to bypass         score 20 -> CAB, site manager and customer notice

Exam traps on operations

• Score change impact from the redundancy state during the work.
• An acknowledged alarm is not a resolved alarm; record and follow up.
• Use only the current controlled version of a procedure.
• Two identity checks before remote hands; a familiar voice is not one.
• Series availabilities multiply; parallel unavailabilities multiply. Do not add percentages.
• Stranded capacity is unusable, not spare.
• Emergency changes still need approval and a record, after the fact if necessary.

Incidents, problems and changes

ITIL separates three practices that exams often mix up. Incident management restores normal service as quickly as possible after an interruption. Problem management finds and removes the underlying causes of incidents, including causes that have not yet produced an incident. Change enablement controls the changes that fix problems and add capacity. A failed CRAH fan is an incident; the discovery that six fans of the same batch have failed in a year is a problem; replacing the remaining fans of that batch is a change. One event can create records in all three.

A known error is a problem whose cause is understood, with a workaround recorded until the permanent fix is in place. When an exam asks where a technician should look first for a quick fix to a recurring alarm, the known error records are the answer.

Measuring operations

Operations teams track a handful of measures, and exams may ask what each one shows. Mean time to acknowledge (MTTA) shows how quickly alarms and tickets are picked up. Mean time to resolve shows how long service takes to restore. SLA attainment is the share of tickets resolved within their targets. Change success rate is the share of changes completed without causing an incident or needing a back-out. Alarm rate per operator compares the desk's load against alarm management guidance. Each measure is only useful if the data behind it is honest, which is why Stonebridge does not punish technicians for raising tickets.

Capacity forecasting

Capacity tracking says where the site is now; forecasting says when it will run out. The simplest forecast takes the current used and committed load, adds the expected growth from customer plans and trends, and finds the month in which the total crosses the usable limit after redundancy. Forecasts must be done separately for each resource, because the first one to run out sets the real limit.

Documentation that operations depend on

Operations rely on accurate documents: as-built drawings that show what was actually installed, single-line diagrams kept current after every electrical change, sequences of operation for cooling and electrical systems, vendor manuals, EOPs, SOPs, MOPs and the shift log. Document control means each has an owner, a version, an approval and a review date, and that superseded versions are withdrawn from the floor. An audit compares documents with the site, and every difference becomes a ticket. Exams often hide the answer to a scenario in a document problem, such as a MOP written against an old one-line.

Standard changes

A standard change is a low-risk, frequent change done to an approved procedure without separate approval each time, such as installing a server in a customer's own cabinet with power already provided, or replacing a failed drive in a hot-swap bay. Standard changes save the CAB's time for the work that needs it. The risk of the category is that people stretch it: a server install that needs a new power circuit is no longer a standard change, because the circuit work touches shared infrastructure. When in doubt, raise it as a normal change and let the approver decide.