Metering, monitoring and control

An electrical system that cannot be seen cannot be operated safely or efficiently. A large data center has thousands of meters, relays, breakers and sensors, and the people running it need to know, at every moment, how much load each part of the system carries, how much redundancy remains, whether power quality is normal, and what happened in the milliseconds before any trip. This lesson describes the electrical power monitoring system (EPMS) that gathers that information: the meters and their accuracy, the metering hierarchy behind PUE, branch circuit monitoring and capacity management, the communication protocols, time synchronization, automated control such as load shedding, and the cybersecurity that must protect it all.

The electrical power monitoring system

An EPMS (sometimes called a power management system or electrical SCADA) collects data from every intelligent device in the electrical system and presents it to operators. A typical architecture has three levels:

• Field devices: revenue-grade and power quality meters at the utility service and generator buses; meters in switchboards, UPS systems and PDUs; microprocessor protective relays (Lesson 3); breaker trip units; branch circuit monitors; and status contacts from transfer switches and breakers.
• Network: dedicated Ethernet networks (usually redundant rings) and gateways that convert older serial protocols, physically separate from the business IT network.
• Servers and software: redundant servers that poll and store the data, display one-line diagrams with live values, raise alarms, analyze power quality events, and produce reports for energy, capacity and billing.

The EPMS is often integrated with, or feeds, the building management system (BMS), which controls mechanical systems, and the data center infrastructure management (DCIM) system, which tracks IT assets, space and capacity.

Meters and accuracy

Electronic power meters sample voltage and current waveforms thousands of times per cycle and compute RMS values, real, reactive and apparent power, energy, power factor, harmonics and demand. Their accuracy is specified by class:

• Revenue-grade meters (for example accuracy class 0.2 or 0.5 under IEC 62053 or ANSI C12.20) at the utility service and wherever energy is billed to tenants: ±0.2% to ±0.5% of reading.
• Power quality meters, which add waveform capture and event recording (Lesson 8).
• General-purpose panel meters and the metering built into UPS systems and PDUs, typically ±1% or so.

A meter is only as accurate as its current transformers (CTs) and voltage connections. CTs have their own accuracy classes, and errors from wrongly sized CTs (operating at a few percent of their rating), reversed polarity or mismatched phases are among the most common metering problems found during commissioning.

Metering for PUE

PUE (DCE-200) is total facility energy divided by IT energy. Its value depends on where the IT energy is measured, so the standard ISO/IEC 30134-2 defines categories:

• Category 1: IT energy measured at the UPS outputs.
• Category 2: measured at the PDU outputs, after PDU transformer and distribution losses.
• Category 3: measured at the IT equipment input, for example at the rack PDUs.

Measuring closer to the IT equipment gives a smaller IT energy figure (losses in between count as facility overhead) and therefore a higher, more honest PUE. Reported PUE values should always state their category and measurement period.

Branch circuit monitoring and capacity management

Branch circuit monitoring (BCM) measures the current in every circuit leaving a PDU or remote power panel, and intelligent rack PDUs measure each outlet or inlet. With thousands of circuits, this data supports three tasks:

• Protecting circuits: alarms before any circuit approaches its continuous rating (80% of the breaker rating for standard breakers, Lesson 11).
• Protecting redundancy: for dual-corded racks, the sum of the A and B circuit loads must stay within what one circuit can carry, because that is what will happen if either path fails (Lesson 5). A rack whose A and B circuits each carry 60% of rating looks healthy but will trip on failover.
• Capacity management: comparing contracted, allocated and measured power at every level, from rack to UPS block to utility service, to find stranded capacity and to stop oversubscription before it becomes risky (Lesson 2).

Communication protocols

Field devices speak a variety of protocols:

• Modbus, in serial (RTU over RS-485) and Ethernet (TCP) forms: simple, universal and still the most common protocol for meters and UPS systems. It has no built-in security.
• IEC 61850: the standard for substation automation, with high-speed peer-to-peer messages (GOOSE) between protective relays that can replace hard-wired trip and interlock signals.
• DNP3: common in utility SCADA, including at the interface with the utility.
• BACnet: the standard for building automation, used where electrical data is shared with the BMS.
• SNMP and Redfish: used by IT-style devices such as rack PDUs and UPS network cards.

Time synchronization and event analysis

When a fault causes a cascade of breaker trips, transfers and alarms within a fraction of a second, the order of events is the key to understanding what happened. That requires every relay, meter and controller to share a common, accurate clock, typically from a GPS-referenced time server distributed by PTP or IRIG-B to within a millisecond or better (DCE-210, Lesson 10). With synchronized timestamps, the EPMS can assemble a sequence-of-events record and waveform captures from many devices into a single timeline, turning hours of guesswork after an incident into minutes of analysis.

Automatic control

Not all control is by operators. The electrical system includes automatic functions that must act faster than any person:

• Transfer: automatic transfer switches and paralleling switchgear move loads between utility and generator power (DCE-340).
• Load shedding: if generator capacity is reduced (a generator fails to start or trips), a load-shedding scheme disconnects lower-priority loads, such as office HVAC, non-critical mechanical plant and, by agreement, some IT load, to protect the generators and the critical load.
• Load add sequencing: loads are added to generators in steps so that each step stays within what the generators can accept.
• Demand response: some facilities reduce grid draw on request from the utility or grid operator, by shifting load to generators or batteries, in exchange for payments or lower tariffs (DCE-420).

Cybersecurity of electrical systems

An EPMS can open and close breakers, change relay settings and start generators. An attacker who gains control of it could cause an outage or damage equipment; attacks on Ukraine's power grid in 2015 and 2016 showed that such attacks are real. Operational technology security practices include:

• Separating the EPMS and BMS networks from IT networks and the internet, with tightly controlled, monitored connections through firewalls and a demilitarized zone.
• Disabling unused services and changing default passwords on every device; many meters and relays ship with well-known defaults.
• Restricting remote access to authenticated, logged sessions through secure gateways.
• Requiring physical or hard-wired confirmation for the most dangerous actions, and leaving protection functions in the relays, independent of the EPMS.
• Following the ISA/IEC 62443 series of standards for industrial control system security.

In the US, CISA publishes guidance for operational technology, utilities follow NERC CIP standards, and federal customers may impose further requirements on the facilities that serve them.