Capstone: tracing a monitoring failure and an interface fault at Eastview

On a Monday morning in March, Donna forwards Sasha two messages. The first is from Piedmont Signal Monitoring: over the weekend, Eastview General Hospital's account showed an IP path failure on Friday evening that never restored, and for 41 minutes early on Sunday the center heard nothing from the hospital at all. The second is from Eastview's facilities manager, who wants the north wing's interfaces checked before the wing's first annual inspection. 'Take both,' Donna says. 'Ravi will back you up, and Gerry signs the report.' This capstone follows Sasha through the week: finding out what happened to the monitoring, fixing it, testing it, and then tracing an interface fault that turns up during the north wing checks.

Monday: what the records say

Sasha starts with the records, before touching anything. She asks the center for the account's event history from Friday to Monday, downloads the event log from Eastview's main node, and asks the hospital's security desk for its log. Laid side by side, they tell a clear story.

# Merge the three logs into one timeline (times as recorded)
center = [("Fri 18:55", "IP path failure received via cellular; security desk called, told 'IT work'"),
          ("Sun 03:22", "no signals or heartbeats from account"),
          ("Sun 04:03", "loss of communication declared; security desk called; on-call technician paged"),
          ("Sun 04:03", "cellular path restored; troubles received"),
          ("Mon 07:30", "IP path still failed")]
panel = [("Fri 18:52", "communicator trouble: IP path 1"),
         ("Sun 03:21", "communicator trouble: cellular path 2"),
         ("Sun 04:02", "cellular path 2 restored")]
desk = [("Fri 18:56", "monitoring call re IP path; IT switch work in progress, logged"),
        ("Sun 04:05", "monitoring call; Bramwell on-call notified")]
order = {"Fri": 0, "Sat": 1, "Sun": 2, "Mon": 3}
key = lambda e: (order[e[0][:3]], e[0][4:])
for t, what, src in sorted([(t, w, "center") for t, w in center] + [(t, w, "panel") for t, w in panel] +
                           [(t, w, "desk") for t, w in desk], key=lambda e: key(e)):
    print(f"{t}  {src:6s} {what}")

Fri 18:52  panel  communicator trouble: IP path 1
Fri 18:55  center IP path failure received via cellular; security desk called, told 'IT work'
Fri 18:56  desk   monitoring call re IP path; IT switch work in progress, logged
Sun 03:21  panel  communicator trouble: cellular path 2
Sun 03:22  center no signals or heartbeats from account
Sun 04:02  panel  cellular path 2 restored
Sun 04:03  center loss of communication declared; security desk called; on-call technician paged
Sun 04:03  center cellular path restored; troubles received
Sun 04:05  desk   monitoring call; Bramwell on-call notified
Mon 07:30  center IP path still failed

On Friday at 18:52 the communicator lost its IP path. It reported the trouble at the panel and sent the path failure to the center over its cellular path, which is exactly what it should do. The center called the security desk, as the account's instructions said. The desk was told it was 'IT work' and logged it. Nobody called Bramwell, so the building ran all weekend on one path. Then at 03:21 on Sunday the cellular path failed too, and for 41 minutes the hospital could not send any signal off the premises.

Two separate questions follow. Why did the IP path fail and stay failed? And why did the cellular path, the backup, fail on Sunday morning?

Monday: the IP path

Tomasz from IT confirms that on Friday evening the hospital replaced the network switches in the main building's communications room. The fire alarm communicator's port was moved onto a new virtual network, and the new firewall policy for that network allowed only traffic the IT team had listed. The communicator's outbound connection to the center's receivers wasn't on the list.

# Compare the communicator's required connections with the new firewall policy (illustrative addresses)
required = [("outbound", "receiver primary", "tcp/443"), ("outbound", "receiver secondary", "tcp/443"),
            ("outbound", "time server", "udp/123")]
policy = [("outbound", "time server", "udp/123"), ("outbound", "hospital proxy", "tcp/8080")]
for rule in required:
    print(f"{rule[0]} to {rule[1]:18s} {rule[2]:8s} {'allowed' if rule in policy else 'BLOCKED'}")

outbound to receiver primary   tcp/443  BLOCKED
outbound to receiver secondary tcp/443  BLOCKED
outbound to time server        udp/123  allowed

The rule that had allowed the communicator to reach its receivers had existed in the old firewall, documented in Bramwell's system records and in an IT ticket from the installation, but it wasn't carried into the new policy. Tomasz adds the two outbound rules, and within a minute the panel's IP path trouble clears and the center receives the restoral.

The deeper cause is a process gap. The hospital's IT change process had no step to check whether a network change affected life safety systems, and the fire alarm's network requirements weren't in the IT team's own records. Sasha and Tomasz agree three changes: the fire alarm's network connections are added to IT's configuration records, any change to those switches or firewall policies requires notice to Bramwell and the center, and the IP path is tested to the center after every such change.

Monday: the cellular path

The cellular path's failure lines up with a planned maintenance window on the carrier's nearest tower, which the carrier had published. During the work, the communicator's traffic would normally move to the next tower. But the communicator's antenna is in the basement electrical room, where Sasha measures the signal from the next-nearest tower at about -107 dBm, below the communicator's minimum. Even the usual tower gives only -101 dBm there, barely above the minimum. The installation had worked for years with almost no margin.

# Signal strength at candidate antenna locations, usual tower and next tower (dBm)
minimum, wanted_margin = -100, 10
sites = {"basement electrical room": (-101, -107), "loading dock wall": (-92, -99), "roof, remote antenna kit": (-77, -84)}
for site, (usual, backup) in sites.items():
    worst = min(usual, backup)
    print(f"{site:26s} usual {usual:5d}  next tower {backup:5d}  worst margin {worst - minimum:3d} dB  "
          f"{'good' if worst - minimum >= wanted_margin else 'not enough'}")

basement electrical room   usual  -101  next tower  -107  worst margin  -7 dB  not enough
loading dock wall          usual   -92  next tower   -99  worst margin   1 dB  not enough
roof, remote antenna kit   usual   -77  next tower   -84  worst margin  16 dB  good

Only the roof gives enough margin from both towers. With the owner's approval, Bramwell installs the communicator maker's listed remote antenna kit, routes the cable in conduit to the roof and re-measures: -77 dBm from the usual tower and -84 dBm from the next one.

Monday: the response gap

The technology failed twice, but the bigger lesson is that a path failure sat for two and a half days without anyone sent to fix it. The account's instructions told the center to call the security desk for any trouble, and the desk had no instruction to call Bramwell. Sasha meets Eastview's facilities manager and the center's operations manager, Marguerite, and they rewrite the instructions:

• any communication path failure: call the security desk and Bramwell's service line; if not restored within 4 hours, Bramwell dispatches a technician;
• loss of all paths: treat as an impairment; call the security desk, Bramwell's on-call technician and, where the authority's rules require it, the fire department; the hospital starts its impairment procedure, including staff watching the on-site workstation;
• the security desk's procedure is updated so that a monitoring call about the fire alarm is never closed as 'IT work' without asking Bramwell.

The changes are written into the account record and the hospital's emergency plan. Marguerite confirms that the corrected arrangement still meets the requirements of the central station service and of NFPA 72 Chapter 26 for the account.

Tuesday: proving the monitoring

With both paths repaired, Sasha tests the monitoring end to end, with the account on test and the center on the phone:

# End-to-end monitoring tests (account on test; times from initiation to operator screen)
limit = 90
tests = [("fire alarm, pull station main lobby", "IP", 6), ("fire alarm, smoke north wing L2", "IP", 7),
         ("supervisory, valve VS-N1", "IP", 6), ("trouble, AC loss RPS-3", "IP", 8),
         ("fire alarm, smoke north wing L2", "cellular (IP disconnected)", 14),
         ("IP path failure", "cellular", 210), ("cellular path failure", "IP", 3480)]
for name, path, s in tests:
    kind = "path" if "path failure" in name else "signal"
    ok = s <= limit if kind == "signal" else True
    print(f"{name:36s} via {path:26s} {s:5d} s  {'ok' if ok else 'too slow'}")

fire alarm, pull station main lobby  via IP                             6 s  ok
fire alarm, smoke north wing L2      via IP                             7 s  ok
supervisory, valve VS-N1             via IP                             6 s  ok
trouble, AC loss RPS-3               via IP                             8 s  ok
fire alarm, smoke north wing L2      via cellular (IP disconnected)    14 s  ok
IP path failure                      via cellular                     210 s  ok
cellular path failure                via IP                          3480 s  ok

Every signal reaches the operator in under 15 seconds on either path, correctly described. Disconnecting the IP path produces a path failure at the center in about three and a half minutes, matching the programmed heartbeat. Disconnecting the cellular antenna produces a cellular path failure in just under an hour, which is within the supervision interval the center and the authority accept for the second path. The new instructions are tested too: the IP path failure call goes to both the security desk and Bramwell's service line.

Wednesday: the north wing interfaces

With the monitoring proven, Sasha moves to the north wing checks, working from the wing's sequence of operation with the hospital's maintenance staff, the elevator mechanic, Joel, and the controls contractor, Kenji. Elevator recall, air handler shutdown and the damper closures all pass. Then a corridor smoke detector on level 2 is operated, and the corridor's fire doors take too long to close.

# Door release times after alarm, north wing level 2 (design maximum 10 s, illustrative)
limit = 10.0
doors = {"L2 corridor east": 14.2, "L2 corridor west": 13.8, "L2 nurse station": 14.0, "L3 corridor east": 1.1}
for d, t in doors.items():
    print(f"{d:18s} {t:5.1f} s  {'ok' if t <= limit else 'SLOW'}")

L2 corridor east    14.2 s  SLOW
L2 corridor west    13.8 s  SLOW
L2 nurse station    14.0 s  SLOW
L3 corridor east     1.1 s  ok

The level 3 doors release in about a second. Level 2's take 14. The doors and holders are identical, and the relays are on the same north wing node, so Sasha looks at the program. The level 3 relays are programmed to follow level 3's detectors directly. The level 2 relays follow a 'smoke compartment group' that someone created on the main node during the wing's last program change. When a level 2 detector operates, its event travels to the main node, the group is evaluated there, and a command travels back to the north wing node to operate the relays.

# Why the round trip is slow during a busy test (illustrative)
hop_ms, hops_each_way = 120, 3
queue_s_per_event = 0.25        # main node handling time per queued event
queued_events = 52              # events from the other test devices still in the queue
round_trip = 2 * hops_each_way * hop_ms / 1000 + queued_events * queue_s_per_event
print(f"network travel {2 * hops_each_way * hop_ms / 1000:.1f} s + queue {queued_events * queue_s_per_event:.1f} s = {round_trip:.1f} s")
print("local control on the north wing node: about 1 s")

network travel 0.7 s + queue 13.0 s = 13.7 s
local control on the north wing node: about 1 s

In a quiet building, the round trip would take under a second. But during an alarm, the main node is processing dozens of events, and the group's command waits its turn. The fault would be worst in exactly the situation that matters: a real fire producing many alarms at once. Ravi agrees the fix: the level 2 door holders are reprogrammed to follow the level 2 smoke compartment's detectors on the north wing node itself, the network group is deleted, and every node is loaded from the corrected database.

Thursday: retests

Sasha retests every function the program change could affect, not just the doors that failed. Every corridor detector on levels 1 to 6 is operated in turn while another technician operates a second group of devices to load the network, as in a real alarm:

# Retest after the program change: release times with the network loaded
results = {"L1": 1.0, "L2": 1.1, "L3": 1.0, "L4": 1.2, "L5": 1.1, "L6": 1.0}
print("all levels within 10 s:", all(t <= 10 for t in results.values()))
print("slowest:", max(results, key=results.get), results[max(results, key=results.get)], "s")

all levels within 10 s: True
slowest: L4 1.2 s

Every door releases in about a second and latches. The elevator recall, fan shutdown and damper tests are repeated because the database change touched every node, and all pass again. The monitoring center receives the alarms from the retests within seconds. The node program versions are recorded, and a copy of the corrected database is placed in the documentation cabinet and Bramwell's files.

Friday: the report

Gerry reviews Sasha's report before it goes to the owner. It covers both problems, their causes and the corrections:

• Monitoring: IP path failed Friday 18:52 after a network change removed the communicator's firewall rules; restored Monday 08:40 by IT. Cellular path failed Sunday 03:21 to 04:02 during carrier maintenance because of low signal margin at the basement antenna; corrected with a listed remote antenna kit on the roof. The building could not send signals off the premises for 41 minutes.
• Response: the IP path failure went unaddressed for over 60 hours because the account's instructions and the security desk's procedure didn't call Bramwell. Instructions, procedure and IT change process revised.
• Interface: north wing level 2 door releases took 14 seconds because they were programmed through a network group on the main node; reprogrammed locally and retested at about 1 second with the network loaded.
• Tests: end-to-end monitoring on both paths, path failure supervision, and every north wing interface; all pass.
• Records: network requirements added to IT's records; account instructions updated; database version recorded; drawings updated for the antenna kit.

Gerry signs it. 'Nothing in this report was a broken detector,' he tells Sasha. 'It was a firewall, a tower, a phone procedure and a line of program. That is what integration work looks like: the fire alarm was fine, and the system still could have failed.'