Commissioning levels, test scripts and documentation

Commissioning a data center involves thousands of individual tests, from checking the torque on a single busbar bolt to failing the utility supply to the whole building. To keep this manageable, the industry organizes testing into levels, each building on the one before: equipment is proven at the factory, then on delivery and installation, then as individual systems, and finally as an integrated facility. This lesson describes the commissioning levels, what happens at each, how test scripts are written and executed, how to test safely around equipment that is being energized for the first time, how many of a set of identical units must be tested, and the documentation that records it all and passes it to the operators.

The five levels

Most data center projects use a five-level scheme. The exact names vary between owners and commissioning firms, but the progression is common:

Each level is a gate. Equipment that fails factory testing is not shipped; a system cannot be functionally tested until its components have passed start-up; integrated testing waits until every system has passed its functional tests and critical issues are closed (Lesson 1). This sequencing means that each level tests something new, rather than rediscovering problems a lower level should have caught.

Level 1: factory witness testing

Large, long-lead equipment is tested at the manufacturer's plant, witnessed by the owner's commissioning team, before it ships. Finding a problem at the factory is far cheaper than finding it on site, where repairs may need specialists flown in and equipment partly dismantled. Typical factory witness tests include running a UPS module at full load on the factory's load bank and through transfers to battery and bypass; testing switchgear wiring, interlocks and protective relays; load testing a generator set across its range; and running a chiller on a test stand at design conditions. The witness checks that the equipment matches the approved submittals (correct ratings, options and firmware), that tests follow the agreed procedure, and that results meet the specification.

Level 2: delivery and installation

On arrival, equipment is inspected for shipping damage (tilt and shock indicators on transformers and UPS cabinets are checked), compared with the order, and stored as the manufacturer requires, often in heated, dry space. After installation, pre-functional checklists verify that it was installed correctly: anchorage and seismic restraints, clearances for maintenance and arc-flash working space, cable sizes and terminations, torque marks on bolted connections, grounding and bonding, piping supports, valve orientation, labeling and access. Many problems that would otherwise surface during functional testing are caught here, such as a reversed flow sensor or a missing ground conductor.

Level 3: start-up and component testing

Level 3 brings each piece of equipment to life on its own. Electrical equipment receives acceptance tests to the NETA ATS standard (Lesson 3): insulation resistance, contact resistance, protective relay calibration, breaker trip tests and transformer turns ratio. Vendors start up their equipment according to their own procedures, and the commissioning team verifies controls point by point, confirming that each sensor reads correctly and each command moves the right device. Records of every result are kept, because they become the baseline for later maintenance testing.

Level 4: functional performance testing

Functional performance tests prove that each system performs all of its functions, at design load and under failure conditions, according to the sequences of operation. For a data center this means testing systems with real heat and electrical load, usually provided by load banks before IT equipment is installed (Lesson 3). A UPS system is tested at full load with a module failed, on battery to verify runtime, and through bypass and maintenance transfers; a chiller plant is tested through staging up and down, failure of a chiller or pump, and restart after a power interruption (Lesson 4). Each test follows a script, and every deviation goes into the issues log.

Level 5: integrated systems testing

Integrated systems testing (IST) is the culmination: the whole facility, under simulated full IT load, is put through realistic failure scenarios, such as complete loss of utility power, to prove that electrical, mechanical, controls, monitoring and fire systems work together and that operators can respond. It is the subject of Lesson 5.

Writing test scripts

A test script (also called a test procedure or functional test) is a step-by-step procedure with expected results. A good script contains:

• Purpose and scope: what the test proves, and which requirement in the OPR it verifies.
• Prerequisites: lower-level tests completed, critical issues closed, documentation available, instruments calibrated.
• Safety: hazards, personal protective equipment, lockout points, arc-flash boundaries and emergency stops; and a statement that the test stops if any step creates an unsafe condition.
• Instruments and resources: load banks, power quality analyzers, thermal cameras, data loggers, radios, and the people and roles needed.
• Initial conditions: the configuration of every relevant breaker, valve and control mode before the test begins.
• Steps: numbered actions, each with an expected result and space to record the actual result and pass or fail.
• Acceptance criteria: measurable limits, such as voltage within ±5 %, transfer time under 10 s, chilled water temperature recovered within 10 minutes.
• Restoration: steps to return the system to normal.
• Sign-off: names, dates and witness signatures.

Scripts are reviewed by the designers, vendors and operators before testing, and executed with a test director calling each step, a recorder writing results, and technicians performing actions. Steps that do not go as expected are stopped, recorded in the issues log, and resumed or rescheduled after the cause is understood. Deviations from the script are never improvised during a live test.

Testing safely

Commissioning is one of the most hazardous periods in a data center's life: equipment is energized for the first time, construction workers are still present, labeling may be incomplete, and systems are deliberately pushed to failure. Safety controls include an energization plan that defines when each part of the system is live and how that is communicated (signs, barriers, daily briefings), strict lockout and tagging, arc-flash studies completed before energization with equipment labeled, qualified electrical workers wearing appropriate protective equipment, and permits for live work only where it cannot be avoided.

In the United States, electrical safety in the workplace follows NFPA 70E and OSHA regulations, including 29 CFR 1910.147 (control of hazardous energy, lockout/tagout) and 1910 Subpart S for electrical work, with construction work covered by 29 CFR 1926.

How many units to test

Large data centers contain hundreds of identical items: rack power distribution units, computer room air handlers, fan coil units, breakers and valves. Owners must decide whether to test every unit fully or a sample. Critical items whose failure could drop IT load are normally tested 100 %, while some functional tests of numerous identical items may be sampled, with the sample expanded if failures are found. Sampling has a statistical cost: small samples can easily miss defects.

from math import comb, ceil, log

def p_detect(N, D, n):
    """Probability that a random sample of n from N units includes at least one of D defective units."""
    return 1 - comb(N - D, n) / comb(N, n)

N = 120                      # identical rack power distribution units on a project
for D in (2, 6, 12):         # number of defective units actually present
    for n in (10, 24, 60, 120):
        print(f"D = {D:2d} defective, sample {n:3d}: P(find at least one) = {p_detect(N, D, n):.2f}")
    print()

# Sample size for 95 percent confidence of catching a defect rate p (large population)
for p in (0.01, 0.05, 0.10):
    print(f"Defect rate {p:.0%}: sample at least {ceil(log(0.05) / log(1 - p))} units")

D =  2 defective, sample  10: P(find at least one) = 0.16
D =  2 defective, sample  24: P(find at least one) = 0.36
D =  2 defective, sample  60: P(find at least one) = 0.75
D =  2 defective, sample 120: P(find at least one) = 1.00

D =  6 defective, sample  10: P(find at least one) = 0.41
D =  6 defective, sample  24: P(find at least one) = 0.75
D =  6 defective, sample  60: P(find at least one) = 0.99
D =  6 defective, sample 120: P(find at least one) = 1.00

D = 12 defective, sample  10: P(find at least one) = 0.67
D = 12 defective, sample  24: P(find at least one) = 0.94
D = 12 defective, sample  60: P(find at least one) = 1.00
D = 12 defective, sample 120: P(find at least one) = 1.00

Defect rate 1%: sample at least 299 units
Defect rate 5%: sample at least 59 units
Defect rate 10%: sample at least 29 units

The results are sobering. If 2 of 120 PDUs are defective, a 20 % sample (24 units) has only a 36 % chance of finding either of them. Even a 50 % sample finds one only 75 % of the time. Sampling works well when defects are systematic, such as a wrong firmware version or a design error affecting every unit, because then a few tests catch them; it works poorly for random installation defects such as a loose connection. That is why installation checks (Level 2) are usually done on every unit even when functional tests are sampled, and why thermal scanning under load, which checks every connection quickly, is a standard part of electrical commissioning (Lesson 3).

Documentation and turnover

The commissioning record must be complete enough that the operators, and auditors, can see exactly what was tested and how the facility should behave. It includes:

• Completed checklists and test scripts with recorded results, signatures and supporting data (power quality recordings, thermal images, trend logs).
• The issues log, with every item closed or formally accepted with a plan.
• As-built drawings and single-line diagrams that match the installed facility, and final protective device settings.
• Operation and maintenance manuals, warranty information and spare parts lists.
• Sequences of operation as finally implemented in the controls.
• Training records for the operations team.
• The commissioning report, summarizing the process, results, outstanding items and recommendations.

These are gathered into turnover packages, often by system, and handed to operations as part of acceptance. Incomplete documentation is one of the most common complaints of operators taking over new facilities; insist on it before accepting handover.