Rack elevations and capacity planning

A cabinet runs out of something long before it runs out of rack units. It may run out of power, of cooling, of weight capacity, of network ports or of room for cables, while half its units are still empty. Capacity planning is the work of keeping track of all of these limits together, so that a customer's next order fits and nothing is overloaded. The detailed planning of halls and power systems belongs to capacity planners and engineers, and the DCE school teaches it. But technicians feed the plan with accurate records and check every installation against it, and a technician who understands the limits catches problems before they are built. Anjali Desai, the site manager, asks Roz to bring Jamal to the monthly capacity review so that he sees how the numbers he records are used.

The rack elevation

A rack elevation is a drawing of a cabinet, unit by unit, showing what is installed in each position. Stonebridge's DCIM system draws elevations from its records: for each device, its name, model, U position, height, weight and power, the power strip outlets it uses and its network connections. The elevation is the first thing a technician checks before an installation, and the last thing updated after it. An elevation that does not match the cabinet is worse than none, because people trust it.

Good elevations follow a few conventions that make cabinets easier to work in:

• Heavy equipment at the bottom. Storage shelves and heavy servers go low, which keeps the cabinet stable and makes lifting safer.
• Switches and patch panels near the top, where cables enter from overhead trays, or near the middle in some designs to shorten cable runs.
• Room left for growth in planned blocks, rather than scattered single units.
• Blanking panels shown explicitly, so that the drawing reminds people to fit them.
• No device straddling a space reserved for something else, such as a future switch.

Space: rack units

Rack units are the easiest limit to see and the least often binding. A 42U cabinet typically loses a few units at the top and bottom to patch panels, switches and cable managers, so perhaps 36 to 38 units are available for servers. Deep and heavy equipment may also leave units that cannot be used because of rail positions or cable bulk.

Power

Power is usually the first limit reached. Each cabinet in Hall A is fed by two rack power strips, A and B, each on its own circuit. Lesson 8 explains the circuits and strips in detail. For planning, the important number is the power the cabinet may draw while keeping its redundancy: if either strip fails, the other must carry the whole cabinet without exceeding its continuous limit.

Electrical codes in both Canada and the United States treat loads that run for three hours or more as continuous, and limit a continuous load on a typical breaker to 80 % of the breaker's rating. Data centre loads are continuous by any definition. So a 30 A circuit is planned for no more than 24 A, and because the cabinet must survive the loss of one side, the whole cabinet, on both strips together, must stay within that 24 A. Stonebridge's planning figure of 40 % of rating per side, from the first course, comes from exactly this: 40 % on each of two sides adds up to 80 % on one side after a failure.

# Power capacity of one cabinet with redundant A and B circuits
import math
volts = 208          # line-to-line voltage of the three-phase rack circuits
breaker = 30         # amperes, each circuit
continuous = 0.80        # continuous loads limited to 80 % of the breaker rating
per_circuit_kw = math.sqrt(3) * volts * breaker * continuous / 1000
print(f"One three-phase {volts} V {breaker} A circuit at 80 %: {per_circuit_kw:.1f} kW")
print(f"Cabinet must survive loss of one side, so plan the whole cabinet for {per_circuit_kw:.1f} kW")
print(f"That is {per_circuit_kw / 2:.1f} kW on each side in normal running (40 % of each circuit's rating)")
planned = 6.2      # kW of equipment already planned for the cabinet
print(f"Already planned: {planned} kW; room left for {per_circuit_kw - planned:.1f} kW")

One three-phase 208 V 30 A circuit at 80 %: 8.6 kW
Cabinet must survive loss of one side, so plan the whole cabinet for 8.6 kW
That is 4.3 kW on each side in normal running (40 % of each circuit's rating)
Already planned: 6.2 kW; room left for 2.4 kW

A cabinet in Hall A on two 208 V, 30 A three-phase circuits can be planned for about 8.6 kW in total. If the customer's servers are listed at their maximum power on the data sheet, that may be only a dozen servers. If they are planned at a measured typical draw, more fit, with the risk that a busy period pushes the load above the plan. Stonebridge plans new installations from the manufacturer's maximum unless the customer has measured data from identical equipment, and then watches the power strip readings, which Lesson 8 explains.

Cooling

Every watt of power a cabinet draws becomes heat that the cooling system must remove, so power and cooling are two views of the same number. Cooling is usually described in kilowatts too, or in BTU per hour in older documents and in the United States. One kilowatt is about 3,412 BTU per hour. Each hall has a cooling capacity per cabinet position, based on its airflow design: Hall A was designed for about 6 kW per position, and a cabinet planned well above that needs its neighbours to be below it, or special measures, or the cabinet does not belong in Hall A at all.

# Heat from a cabinet, in the units the cooling team uses
cabinets_kw = {"A-31-01": 5.8, "A-31-02": 7.9, "A-31-03": 6.2, "A-31-04": 3.1}
design_per_position = 6.0   # kW per position in Hall A
for cab, kw in cabinets_kw.items():
    btu = kw * 3412
    flag = "" if kw <= design_per_position else "  above design: check neighbours"
    print(f"{cab}: {kw:4.1f} kW = {btu:7,.0f} BTU/h{flag}")
total = sum(cabinets_kw.values())
print(f"Cage total {total:.1f} kW against {design_per_position * len(cabinets_kw):.1f} kW design for {len(cabinets_kw)} positions")

A-31-01:  5.8 kW =  19,790 BTU/h
A-31-02:  7.9 kW =  26,955 BTU/h  above design: check neighbours
A-31-03:  6.2 kW =  21,154 BTU/h  above design: check neighbours
A-31-04:  3.1 kW =  10,577 BTU/h
Cage total 23.0 kW against 24.0 kW design for 4 positions

In the example, one cabinet is above Hall A's design figure, but the cage as a whole is within its total because one cabinet is lightly loaded. The cooling team accepts that, as long as the heavy cabinet is not next to another heavy one and its blanking and containment are perfect. If the customer later fills the light cabinet, the cage will be over.

Weight

Lesson 2 covered the cabinet's own ratings. Capacity planning adds the floor. In Hall A each cabinet stands on raised floor tiles and pedestals with their own ratings, so a cabinet that is within its own rating can still be too heavy for its position. Stonebridge's DCIM records each position's allowable load, and the elevation's weight total is checked against the lower of the cabinet rating and the floor rating.

Network ports and cables

The fourth limit is connectivity. A cabinet's top-of-rack switches have a fixed number of ports, and its patch panels a fixed number of positions. Every server needs a certain number of connections, often two data ports and one management port. And every cable takes space in the cable managers and adds weight. A cabinet with all its units full but its cable managers overflowing is a cabinet nobody can work in, and it blocks airflow at the rear.

Putting the limits together

At the capacity review, the DCIM report shows each cabinet's use of space, power, weight and ports as a percentage of its limit, and the planners look for whichever is highest. Anjali calls it the first wall: the limit a cabinet will hit first. For most of Hall A's enterprise cabinets the first wall is power. For a few storage-heavy cabinets it is weight. Almost never is it rack units.

# Which limit will this cabinet hit first?
limits = {  # resource: (planned use, limit)
    "rack units": (28, 38),
    "power kW": (7.4, 8.6),
    "weight kg": (820, 1088),
    "network ports": (40, 48),
}
for name, (use, lim) in limits.items():
    print(f"{name:14s} {100 * use / lim:5.1f} % used")
first = max(limits, key=lambda k: limits[k][0] / limits[k][1])
print("First wall:", first)

rack units      73.7 % used
power kW        86.0 % used
weight kg       75.4 % used
network ports   83.3 % used
First wall: power kW

This cabinet still has ten rack units free, but it has less than 14 % of its power left. A customer looking at the cabinet would think it half empty; the elevation and the percentages say it is nearly full.

What technicians feed into the plan

The capacity plan is only as good as the records behind it, and most of those records are made by technicians. Every installation and removal must update DCIM with the device, its position, its power connections and its weight. Every power strip reading that looks odd must be reported. Every cabinet found to differ from its elevation must be corrected. Anjali shows Jamal two cabinets in which DCIM was wrong last quarter: one had an unrecorded server drawing 600 W, the other recorded a removed array that was still there. Both were found by technicians on unrelated tickets, who noticed and reported the difference.

Reserving space

Customers often want space held for future equipment. DCIM lets the planners reserve units, power and ports in a cabinet for a named future installation, so that nobody else uses them in the meantime. Reserved units are fitted with blanking panels like any empty unit, and they appear on the elevation in a different colour so that technicians do not fill them by mistake. Reservations have dates, and the capacity team reviews them each quarter, releasing ones the customer no longer needs. Jamal finds that a third of the apparently free space in one Hall A cage is reserved for a customer's planned expansion, which explains why a request from a different team in the same company was declined.

At the hall level the same thinking applies. The capacity team tracks how much of each hall's power, cooling and space is sold, reserved and actually used, because customers rarely use all the power they buy. The difference between sold and used power is part of how a colocation business plans new customers, and it is one more reason that accurate measurements from technicians matter.