Cleaning and inspecting high-density connectors

Nadia keeps a framed printout on the wall of the cabling workshop at YUL-2: an inspection-scope image of an MPO-16 end face with a fingerprint smeared across eleven of its sixteen fibers. It came from a Hall D link that had been dropping packets for a week, slowing one of Corvid's training jobs. The fix took ninety seconds with a cleaning tool. LVT-210 taught Jamal to clean and inspect duplex connectors; this lesson is about doing it with multi-fiber connectors, in bulk, in a live hall, where one dirty face can affect a sixteen-fiber link and a careless cleaning routine can spread contamination to hundreds.

Why contamination matters more here

A speck of dust a few microns across, invisible to the eye, can cover much of a single-mode core, which is only about nine microns wide. On a duplex connector that means one link with high loss or reflections. On an MPO connector, the end face is a row of twelve or sixteen fibers in a wide ferrule, and dust tends to collect across it, so several fibers are affected at once. Worse, the large flat face of an MPO ferrule and its guide pins pick up debris from the dust cap and the adapter, and when two dirty MPO faces are pushed together, the debris is pressed into the glass and can scratch or pit several fibers permanently.

High-speed parallel links are also less tolerant than the 10 Gb/s duplex links of Hall A. Their loss budgets are smaller, as Lesson 11 shows, and their modulation, especially the four-level PAM4 signaling used at 100 Gb/s per lane and above, is more sensitive to reflections and noise. A contamination that a 10G link would shrug off can push a 100G lane into errors.

Inspect, clean, inspect

The rule from LVT-210 does not change: inspect before you connect. If it is clean, connect it. If it is dirty, clean it and inspect again. Never clean blindly and connect without looking, because cleaning can move dirt as well as remove it, and never assume a connector fresh from its bag is clean, because dust caps are a common source of contamination. In the data center the rule applies to both sides of every mated pair: the cord and the adapter in the panel, or the cord and the transceiver port.

Scopes for MPO

Inspecting a duplex connector shows one fiber in a small field of view. An MPO face needs a different approach. A video inspection probe with an MPO tip and a wide field of view first shows the whole ferrule face, so that loose debris on the face and near the pin holes can be seen; then the probe scans or zooms through each fiber position to inspect the cores in detail. Automated probes step through all twelve or sixteen positions and grade each one, which turns an inspection that would take several minutes by hand into one that takes seconds, and produces a saved record.

The grading follows IEC 61300-3-35. The 2022 edition of that standard changed several things that matter in practice. It added a wide-field check of the whole ferrule face for multi-fiber connectors, because loose debris there causes contamination when the connector is mated. It stopped grading the adhesive and contact zones of the end face separately as pass or fail, and it slightly reduced the area that must be clean around each core. It also accepts that particles that cannot be removed after several cleanings, if they are outside the core and do not affect performance, need not make the connector fail. The core zone remains strict: nothing on the core. Stonebridge's probes are set to the current criteria, and their results are saved with the link record.

Cleaning tools for MPO

MPO connectors need their own cleaning tools, because a duplex cleaner cannot reach across the ferrule:

• Push-type MPO cleaners, which look like a large pen, have a fresh strip of cleaning tape inside. Pushed into an adapter or onto a connector, they wipe the whole face in one stroke. They come in versions for pinned and unpinned connectors and for MPO-12, MPO-16 and MPO-24.
• Cassette or reel cleaners have a strip of fabric that advances for each wipe, for cleaning a loose connector held in the hand.
• Wet-dry cleaning uses a small amount of fast-drying optical solvent on a lint-free wipe or swab, then a dry wipe, for contamination that dry cleaning does not move, such as oil from fingers or residue from a cap.

Each tool is good for a certain number of cleans, and a worn-out cleaner spreads dirt. Stonebridge's technicians carry a counter tally on their carts and replace cleaners at the maker's limit. The solvent is in small sealed dispensers, and wipes are never reused.

Cleaning in bulk

Installing a pod in Hall D means mating thousands of MPO end faces in a few days. Nadia organizes it as a production line:

• Trunks arrive with caps on and the factory test report; the caps stay on until the leg is plugged in.
• The panel side is inspected first, every adapter, and cleaned if needed, before any cord goes in.
• Each cord end is inspected out of the bag, cleaned if needed, inspected again and mated immediately, then the cap is put into a bin, not back on the cord.
• The automated probe saves an image and a grade for every end face, named by its port label.
• At the end of the shift, the inspection results are compared with the port map so that no end face was missed.

# Summarize a shift's inspection log for MPO end faces
log = [("D-4-02-PP3-01 rear", "pass", 0), ("D-4-02-PP3-02 rear", "fail", 1), ("D-4-02-PP3-03 rear", "pass", 0),
       ("cord C-1182 end A", "fail", 1), ("cord C-1182 end B", "pass", 0), ("D-4-02-PP3-04 rear", "fail", 2),
       ("cord C-1183 end A", "pass", 0), ("cord C-1183 end B", "fail", 3)]
first_pass = sum(1 for _, r, c in log if c == 0)
fixed = sum(1 for _, r, c in log if r == "fail" and 0 < c <= 2)
stubborn = [n for n, r, c in log if c > 2]
print(f"{len(log)} end faces: {first_pass} passed first time, {fixed} passed after cleaning")
print(f"First-time pass rate {first_pass / len(log):.0%}")
for n in stubborn:
    print(f"  {n}: still failing after {3} cleanings -> inspect for damage, replace or report")

8 end faces: 4 passed first time, 3 passed after cleaning
First-time pass rate 50%
  cord C-1183 end B: still failing after 3 cleanings -> inspect for damage, replace or report

A first-time pass rate well below Stonebridge's usual figure points to a supply problem, such as dirty caps or a contaminated batch of cords, and Nadia wants to know about it the same day.

Transceiver ports and live links

The receptacle in a transceiver is the hardest place to inspect and clean, because the probe must reach into the port, and the transceiver may be in a live switch next to working links. Stonebridge's rules are that a technician never inspects or cleans a port that is part of a working link unless the ticket says the link has been taken down, that the transceiver's laser is treated as live even when the link is down, and that the probe used must have a screen or filter so that no one looks directly into a port. Customer transceivers are cleaned only with the customer's agreement, because some makers require their own cleaning tools.

Damaged connectors are replaced, not reused. A pit or scratch on a core will not clean off and will cause high loss or reflections for as long as it is in service. A damaged cord goes in the bin, a damaged adapter is replaced, and a damaged trunk leg is reported so that the trunk's warranty can be used.

Keeping the tools clean

Inspection probes and cleaners can carry dirt as well as remove it. Probe tips are cleaned and checked at the start of each shift, test reference cords are inspected before every use, and cleaning tools are kept in their cases, not loose on a cart where they pick up dust. Dust caps that come off connectors are thrown away, not kept in a pocket and pressed back on later, because a cap that has been in a pocket is dirtier than the air of the hall. Stonebridge's carts carry a sealed bin for used caps and wipes, a supply of fresh caps for connectors that must be capped again, and a log sheet or tablet for the inspection records.

Recording inspections

On large installations, and for customers whose contracts require it, every end face is inspected with an automated probe that saves an image and a pass or fail grade named by the port or cord label. The images go into the handover package and the link's record. They are useful long after the installation: when a link develops errors months later, comparing a new image with the one taken at installation shows whether the end face has changed, which tells the technician whether the problem is a new contamination or an old defect. Corvid's specification requires saved images for every backend end face in Hall D, and Stonebridge now saves them for every new trunk it installs, whatever the customer.

# Find end faces that were never inspected, by comparing the port map with the probe's saved results
port_map = ["D-4-02-PP3-37", "D-4-02-PP3-38", "D-4-02-PP3-39", "D-4-02-PP3-40", "D-4-02-PP3-41"]
saved = {"D-4-02-PP3-37": "pass", "D-4-02-PP3-38": "pass", "D-4-02-PP3-40": "pass after cleaning", "D-4-02-PP3-41": "pass"}
missing = [p for p in port_map if p not in saved]
print(f"{len(port_map)} ports, {len(saved)} with saved results")
for p in missing:
    print(f"  no inspection record for {p}: inspect before the ticket closes")

5 ports, 4 with saved results
  no inspection record for D-4-02-PP3-39: inspect before the ticket closes

Nadia also keeps a short list of the contamination sources she has found most often in Stonebridge's halls: dust caps that were reused, cords laid on the floor or the top of a cabinet while their far end was connected, fingers that touched an end face while a latch was being released, and dust from building work such as ceiling or floor tile replacement. Most of them are prevented by habits rather than tools, which is why the same habits are drilled into every new technician.