Capstone: cabling a GPU pod in Hall D

In DCT-130, Jamal helped install rack D-3-07, the first of eight integrated GPU racks for Corvid AI's expansion in Hall D. Now the racks are in, D-3-08 has come back from the integrator with its coolant bracket repaired, and the D-3-11 power mix-up is long fixed. What the eight racks need before they can train anything is their cabling: the backend fabric that joins their GPUs, the frontend links to storage and the outside world, and the management network. Nadia leads the cabling project for Stonebridge. Carmen Ortiz represents Corvid, which owns the switches and transceivers and supplies the port maps. Jamal is on the installation crew from start to finish, and this lesson follows the project through his notebook.

The design

Corvid's design follows the rail-optimized shape from Lesson 3. Each rack has 18 compute trays with four GPUs each, and every GPU has one 800 Gb/s backend port with a single-mode MPO-16 transceiver. GPU 1 of every tray connects to rail 1, GPU 2 to rail 2 and so on. The leaf switches for rails 1 to 4 sit in network cabinets D-4-01 to D-4-04 in the next row, and the spine switches in D-4-05 to D-4-08. Frontend links run on twin-port 2×400G modules with duplex LC APC connectors through cassettes to the frontend network cabinet D-4-09, and the management network runs on Category 6A to a management switch in each rack.

Counting the job

Nadia starts every project with the counts, because they decide the material order, the tray plan and the crew's schedule.

# Material counts for the Corvid pod (eight racks)
racks, trays, rails = 8, 18, 4
legs_per_trunk = 9                     # 144-strand base-16 trunk, MPO-16 legs
gpu_links = racks * trays * rails
per_rack_per_rail = trays
trunks_rack_rail = -(-per_rack_per_rail // legs_per_trunk)
backend_trunks = racks * rails * trunks_rack_rail
leaves = rails * 5
leaf_spine_links = leaves * 32
leaf_spine_trunks = -(-leaf_spine_links // legs_per_trunk)
frontend = racks * trays
print(f"GPU backend links {gpu_links}: {backend_trunks} trunks ({trunks_rack_rail} per rack per rail)")
print(f"Leaf-to-spine links {leaf_spine_links}: {leaf_spine_trunks} trunks")
print(f"Frontend duplex links {frontend}: {-(-frontend // 6)} cassettes at each end (6 duplex ports each)")
mated = 2 * (gpu_links + leaf_spine_links)
print(f"MPO-16 cords to install {mated:,}; MPO end faces to inspect at least {mated * 2 + (gpu_links + leaf_spine_links) * 2:,}")

GPU backend links 576: 64 trunks (2 per rack per rail)
Leaf-to-spine links 640: 72 trunks
Frontend duplex links 144: 24 cassettes at each end (6 duplex ports each)
MPO-16 cords to install 2,432; MPO end faces to inspect at least 7,296

The last line is the one Jamal underlines. Every link has a cord at each end, so every link brings four cord end faces and the two panel ports they plug into, and every one of them is inspected before mating. More than seven thousand inspections is why the inspection line from Lesson 8 had to be organized like a factory.

Planning the route

The trunks run from the top of each GPU rack, up to the trunk tray above row D-3, along the tray and across to row D-4, then down through waterfalls into the top of each network cabinet. Jamal's fill check in Lesson 10 had found old commissioning copper in the tray above D-3-02; it was removed before the project began, and the tray's fill after the installation is planned at 41 %. The trunk lengths are worked out from the route and ordered with a small allowance, because factory trunks cannot be shortened.

# Trunk length for each GPU rack to each rail cabinet, from the route (planning)
rack_pos = {"D-3-05": 0, "D-3-06": 1, "D-3-07": 2, "D-3-08": 3, "D-3-09": 4, "D-3-10": 5, "D-3-11": 6, "D-3-12": 7}
cab_pos = {"D-4-01": 0, "D-4-02": 1, "D-4-03": 2, "D-4-04": 3}
pitch, row_gap, drops, slack = 0.8, 4.2, 6.0, 2.0
stock_step = 1
lengths = []
for r, rp in rack_pos.items():
    for c, cp in cab_pos.items():
        run = abs(rp - cp) * pitch + row_gap + drops + slack
        lengths.append(-(-run // stock_step) * stock_step)
print(f"{len(lengths)} rack-to-rail routes; shortest {min(lengths):.0f} m, longest {max(lengths):.0f} m")
print("Order lengths:", sorted(set(lengths)))

32 rack-to-rail routes; shortest 13 m, longest 18 m
Order lengths: [13.0, 14.0, 15.0, 16.0, 17.0, 18.0]

Every route is well inside the 500 m (1,640 ft) reach of the DR8 optics, so the only real limit is the loss budget, which the design checks next.

The loss budget

Each backend link has a cord from the GPU's transceiver to the pass-through panel at the top of its rack, a trunk to the rail cabinet, and a cord to the leaf switch's transceiver. The two mated pairs at the panels count against the channel budget of about 3.0 dB. Corvid's specification calls for low-loss APC MPO-16 connectors with a maximum of 0.35 dB per mated pair and requires every link to show at least 1.0 dB of margin.

# Worst-case planned loss of a backend link against the DR8 budget
budget = 3.0
length_km = 0.019
fibre_db = length_km * 0.4          # single-mode at 1310 nm, typical worst case per km
pairs = 2
per_pair = 0.35
planned = fibre_db + pairs * per_pair
print(f"Fibre {fibre_db:.3f} dB + {pairs} mated pairs {pairs * per_pair:.2f} dB = {planned:.2f} dB planned")
print(f"Margin against {budget} dB budget: {budget - planned:.2f} dB  (Corvid requires at least 1.0 dB)")

Fibre 0.008 dB + 2 mated pairs 0.70 dB = 0.71 dB planned
Margin against 3.0 dB budget: 2.29 dB  (Corvid requires at least 1.0 dB)

Installation

The installation runs over two weeks of night and day shifts, scheduled around Corvid's work in the hall. Jamal's notebook records the stages:

• Receiving. Tomás Reyes checks in 64 backend trunks and 72 leaf-to-spine trunks against the order, with their factory reports, and quarantines two whose reels show crush damage.
• Panels. The crew installs pass-through MPO-16 panels at the top of each GPU rack and in the network cabinets, and the frontend cassettes. Every rear adapter is inspected before a trunk leg goes in.
• Trunks. Trunks are laid into the tray, not pulled, two crew members feeding and one in the lift guiding each trunk through the waterfalls. Each trunk is labelled at both ends and at every change of direction, as Lesson 11 described.
• Termination. The legs are plugged into the rear of the panels in leg order, matched to the port map, each end face inspected and cleaned first. Spare legs are capped and stowed.
• Testing. Every installed permanent link is tested with the MPO light source and power meter at 1310 nm, giving per-fibre loss and the position map.
• Cords. With Corvid's engineers present, Stonebridge installs the MPO-16 cords from the panels to the transceivers, one tray at a time, while Corvid checks each link on the switches.

What the tests found

# Summary of the backend permanent-link tests and the follow-up
results = {"links tested": 576 + 640, "passed first time": 1204, "high loss, fixed by cleaning": 9,
           "high loss, pinched leg re-dressed": 1, "trunk legs replaced": 2, "polarity faults": 0}
total = results["links tested"]
print(f"{total} permanent links tested; first-time pass rate {results['passed first time'] / total:.1%}")
for k, v in results.items():
    if k not in ("links tested", "passed first time"):
        print(f"  {k}: {v}")
fixed = total - results["passed first time"]
print(f"All {fixed} failures resolved and re-tested; final pass rate 100%")

1216 permanent links tested; first-time pass rate 99.0%
  high loss, fixed by cleaning: 9
  high loss, pinched leg re-dressed: 1
  trunk legs replaced: 2
  polarity faults: 0
All 12 failures resolved and re-tested; final pass rate 100%

Three findings stand out in Jamal's notes. Nine links failed because of dirt on panel adapters that had been inspected before termination; the crew traced it to dust from a ceiling tile replaced during the night shift, and inspection of that panel was repeated. One leg, TR-D-0417 leg 6, showed 3.2 dB on one fibre: the leg had been pinched under the tray's waterfall cover, and once re-dressed it measured 0.7 dB. Two legs had cracked connectors from the damaged reels Tomás had quarantined; a crew member had used them by mistake, and they were replaced with legs from the spare trunk.

The polarity checks found no faults, because every trunk, panel and cord was of the Method B type Stonebridge uses throughout. But the final audit against Corvid's port map did find a problem that the tests could not: two trunks from rack D-3-09 had been terminated in each other's rail cabinets, rail 2's trunk in D-4-03 and rail 3's in D-4-02. Every fibre tested perfectly, and the links would have come up, but on the wrong rails. Jamal caught it because he compared the trunk labels at both ends with the port map before Corvid connected the cords. The two trunks were swapped at the network end, re-tested and re-labelled the same night.

Handover

The handover package goes to Carmen at the end of the second week: the as-built tray drawings, the trunk schedule with serial numbers, the port maps for all four networks, the factory and field test results for every link, the saved inspection images for every end face, as Corvid's specification required, and the label list. Carmen's team reviews it over two days, brings up all 1,216 backend and leaf-to-spine links, and runs a test training job across the pod. The switches show every GPU on its correct rail and no errors on any link.

Nadia asks Jamal what he would do differently next time. He says he would schedule the ceiling work and the termination on different shifts, and that he would compare the trunk labels with the port map as each trunk was terminated, not at the end, so that a swap is caught when it costs five minutes rather than an evening.

Working alongside the tenant

Throughout the project, Corvid's engineers worked in the same hall on their own equipment, and the two teams had to share the space without getting in each other's way. Nadia and Carmen met at the start of every shift to agree which racks and cabinets each team would work in, and no Stonebridge technician touched a transceiver, switch or tray inside an integrated rack without a Corvid engineer present. Where the two teams' work met, at the cords between the panels and the transceivers, each cord was installed by Stonebridge and checked by Corvid on the switch before the next one went in. Jamal found that the slowest part of the job was not the cabling itself but the handoffs, and that a clear shared list of who was doing what on each shift saved more time than any tool.