Capstone: a flapping uplink in Hall D

On a Tuesday in Jamal's last month of the networking term, Corvid's monitoring raises a ticket at 14:05: the uplink from leaf d4-leaf-r3-02 to spine d4-spine-03 has gone down and up eleven times since the morning. The link is one of the 800 Gb/s leaf-to-spine links of the rail 3 backend fabric for the pod in row D-3. Because the fabric has many uplinks, no job has failed, but each flap moves traffic off and back on, and Corvid's engineers have seen the pod's training step time rise slightly on every flap. Carmen Ortiz assigns the ticket to Stonebridge's floor team with herself as contact. Roz Kerr lets Jamal lead it and stays at his shoulder. This capstone follows the job from the ticket to the soak, using every lesson of the course.

1. Read the ticket and confirm the target

The ticket gives the two ends: d4-leaf-r3-02 port Ethernet1/57 in cabinet D-4-03 at U33, and d4-spine-03 port Ethernet1/12 in cabinet D-4-06 at U36. Both optics are 800G-DR8 OSFP modules, eight lanes of single-mode fiber in each direction, joined by an MPO-16 trunk leg through a pass-through panel at each cabinet and an MPO-16 patch cord at each end, as DCT-210 Lesson 12 recorded when the pod was cabled. Jamal checks the port map and the trunk record: trunk TR-D-0522, leg 3. He asks Carmen to drain the link, as Lesson 6 described, and waits for her word before touching anything. At 14:20 she confirms it is drained.

2. Gather the evidence

Before going to the cabinets, Jamal asks Carmen for the switch's view at both ends, because the readings will tell him where to look. Carmen pastes the link history, the counters and the optic readings into the ticket.

# Optic readings at both ends of the flapping link (dBm per lane), from the ticket
leaf_tx  = [1.1, 1.2, 1.0, 1.3, 1.2, 1.1, 1.2, 1.1]   # d4-leaf-r3-02 Eth1/57 transmit
spine_rx = [0.3, 0.4, 0.2, 0.5, 0.3, 0.2, -6.9, 0.4]  # d4-spine-03 Eth1/12 receive
spine_tx = [1.2, 1.1, 1.3, 1.2, 1.0, 1.2, 1.1, 1.3]
leaf_rx  = [0.4, 0.3, 0.5, 0.4, 0.2, 0.4, 0.3, 0.5]
loss = 0.8   # measured loss of the path at installation (DCT-210 records)
for name, tx, rx in (("leaf -> spine", leaf_tx, spine_rx), ("spine -> leaf", spine_tx, leaf_rx)):
    worst = max(range(8), key=lambda i: (tx[i] - loss) - rx[i])
    extra = (tx[worst] - loss) - rx[worst]
    print(f"{name}: worst lane {worst + 1}, {extra:.1f} dB more loss than recorded")

leaf -> spine: worst lane 7, 7.3 dB more loss than recorded
spine -> leaf: worst lane 2, 0.0 dB more loss than recorded

The direction from leaf to spine has a problem on lane 7, about 7 dB more loss than the path had when it was installed; the other direction is perfect. The temperatures and bias currents at both ends are normal, so neither laser is failing. Following Lesson 11, the evidence points to something in lane 7's fiber path from the leaf's transmitter to the spine's receiver: a connector, a cord or the trunk. The link history shows something else worth noticing: the flaps are not evenly spread. They come in clusters.

3. Look for a pattern

A link that flaps in clusters is often being disturbed. Jamal asks Dmitri Volkov, the shift lead, for the door sensor log of the two cabinets from the DCIM system, which records every time a cabinet door is opened.

# Do the link flaps line up with cabinet door openings? (times as minutes after 06:00)
flaps = [92, 93, 211, 212, 214, 330, 331, 401, 402, 470, 471]
doors = {"D-4-03 front": [90, 209, 468], "D-4-06 rear": [329, 399], "D-4-05 front": [150, 290]}
window = 4   # minutes after a door event
for cab, events in doors.items():
    matched = [f for f in flaps if any(0 <= f - e <= window for e in events)]
    print(f"{cab:13} {len(events)} openings, {len(matched)} of {len(flaps)} flaps within {window} min after")
explained = {f for f in flaps for ev in doors.values() for e in ev if 0 <= f - e <= window}
print(f"Flaps explained by door activity: {len(explained)} of {len(flaps)}")

D-4-03 front  3 openings, 6 of 11 flaps within 4 min after
D-4-06 rear   2 openings, 4 of 11 flaps within 4 min after
D-4-05 front  2 openings, 0 of 11 flaps within 4 min after
Flaps explained by door activity: 10 of 11

Every flap followed within a few minutes of a door opening on one of the two cabinets that hold the link's ends: the front door of the leaf cabinet D-4-03, and the rear door of the spine cabinet D-4-06. Corvid's own staff had been in and out of both all day fitting blanks after the cage-blank finding of Lesson 8. Something in lane 7's path moves when a door moves.

4. Inspect at the cabinets

Jamal starts at the spine end, where the errors are counted. In cabinet D-4-06 he opens the rear door slowly and watches the cords. The spine's MPO-16 patch cord from Ethernet1/12 runs down the vertical manager to the pass-through panel; it is fine. He closes the door and checks the panel's front: the cord is in, and the door clears it easily. He inspects the cord end at the spine optic and the optic's port, as Lesson 8 requires: both clean. He inspects the trunk leg end behind the panel: clean.

At the leaf end, in cabinet D-4-03, the patch cord from Ethernet1/57 to the panel is longer than the others in the row and has been looped in front of the vertical manager instead of inside it. When Jamal closes the front door slowly, the door's inner edge presses the loop against the panel, and the cord's MPO-16 connector at the panel tilts. He opens the door again and presses the connector gently: it moves. It was never fully latched. It made contact well enough to pass light on most lanes, but lane 7, near the edge of the 16-fiber array, lost contact whenever the connector tilted.

5. Fix one thing, and check

Jamal tells Carmen what he has found and proposes the fix: disconnect the cord at the panel, inspect and clean both the cord end and the panel port, reconnect until it clicks, and re-route the cord inside the vertical manager with its slack stored properly, as DCT-210 Lesson 10 describes. Carmen agrees. He inspects the cord end: two particles on the end face, one near fiber 7, almost certainly from being disturbed so often. He cleans it, reinspects it clean and saves the image; the panel port is clean. He reconnects with a firm push until the latch clicks, pulls gently to confirm it holds, and routes the cord through the manager. He then opens and closes the door three times and watches the connector: it does not move.

# Lane 7 received power at the spine, before and after the fix
before = [0.3, 0.4, 0.2, 0.5, 0.3, 0.2, -6.9, 0.4]
after  = [0.3, 0.4, 0.2, 0.5, 0.3, 0.3, 0.2, 0.4]
for name, rx in (("before", before), ("after", after)):
    spread = max(rx) - min(rx)
    print(f"{name:6} lanes {min(rx):+.1f} to {max(rx):+.1f} dBm, spread {spread:.1f} dB -> {'ok' if spread <= 1.5 else 'one lane low'}")

before lanes -6.9 to +0.5 dBm, spread 7.4 dB -> one lane low
after  lanes +0.2 to +0.5 dBm, spread 0.3 dB -> ok

With the link still drained, Carmen reads the counters and the optic readings: lane 7 at the spine now receives as much light as the others, the link is up at 800 Gb/s with RS544 FEC at both ends, and the BGP session is established. She asks for a door test with the readings live: Jamal opens and closes the front of D-4-03 and the rear of D-4-06 twice each while she watches. Nothing flickers.

6. Soak and hand back

Corvid's acceptance rule for a repaired backend link, from Lesson 9, asks for optic readings within warnings on every lane, received power within a narrow window across lanes, no uncorrected FEC errors and corrected errors below a set rate over a soak period, with no flaps. Carmen soaks the link for thirty minutes drained, then undrains it and keeps the ticket open overnight. The next morning the link has carried traffic for eighteen hours, through two dozen door openings in the row, with no flaps and corrected errors at the level of its neighbors.

# Port turn-up checklist for a repaired fabric link, and the ticket note it produces
result = {
    "ports confirmed against ticket and labels": True,
    "both ends inspected, images saved": True,
    "connectors latched (pull test)": True,
    "cords routed in managers, clear of doors": True,
    "link up at 800G, FEC RS544 both ends": True,
    "BGP session established with expected neighbor": True,
    "all lanes within warnings, spread <= 1.5 dB": True,
    "no uncorrected FEC, no flaps in 18 h soak": True,
}
for item, ok in result.items():
    print(("[x] " if ok else "[ ] ") + item)
note = ("RH-52944 d4-leaf-r3-02 Eth1/57 <-> d4-spine-03 Eth1/12 (TR-D-0522 leg 3). "
        "Cause: leaf-side MPO-16 cord at D-4-03 panel not fully latched; door pressed looped cord and tilted "
        "connector, lane 7 lost contact on door movement (11 flaps matched to door log). "
        "Action: cleaned cord end (2 particles), reconnected and latched, re-routed in manager, door test x2. "
        "Spine lane 7 Rx -6.9 -> +0.2 dBm. Soak 18 h: no flaps, no uncorrected FEC. No parts replaced.")
print()
print("Ticket note:", note if all(result.values()) else "checklist incomplete, do not close")

[x] ports confirmed against ticket and labels
[x] both ends inspected, images saved
[x] connectors latched (pull test)
[x] cords routed in managers, clear of doors
[x] link up at 800G, FEC RS544 both ends
[x] BGP session established with expected neighbor
[x] all lanes within warnings, spread <= 1.5 dB
[x] no uncorrected FEC, no flaps in 18 h soak

Ticket note: RH-52944 d4-leaf-r3-02 Eth1/57 <-> d4-spine-03 Eth1/12 (TR-D-0522 leg 3). Cause: leaf-side MPO-16 cord at D-4-03 panel not fully latched; door pressed looped cord and tilted connector, lane 7 lost contact on door movement (11 flaps matched to door log). Action: cleaned cord end (2 particles), reconnected and latched, re-routed in manager, door test x2. Spine lane 7 Rx -6.9 -> +0.2 dBm. Soak 18 h: no flaps, no uncorrected FEC. No parts replaced.

Carmen closes the ticket with Jamal's note. Two follow-ups come out of it. Nadia Okoro, the campus cabling lead, checks every leaf and spine cabinet in rows D-3 and D-4 for cords looped outside their managers and finds four more, none yet causing errors; she schedules their re-routing under one ticket with Corvid. And Corvid adds a door-event field to its link-flap alarms, so that the next flapping link is compared with the DCIM door log automatically.

Looking back

The job used the whole course. Lesson 1's habits set it up: the right owner, the right ports, a contact watching, nothing touched outside the ticket. Lesson 6's drain made the work a planned event in a live fabric. Lessons 7 and 8 explained the optic and its readings, and Lesson 9 why one lane in one link mattered to a whole pod. Lesson 11's method turned the readings into a direction, lane 7 from leaf to spine, and kept the fix to one change. And the pattern in the door log, which no reading on its own would have shown, came from treating the fault as a physical event with a cause in the room. Roz's comment on the ticket was one line: right cord, right reason, one change.