Telecommunications facilities, power and timing

Data centers grew up alongside an older kind of building: the telephone company central office. Telecommunications facilities have their own engineering traditions, many of which still shape data center practice: −48 V DC power plants with hours of battery reserve, equipment standards that demand survival through earthquakes and fires, and an obsession with precise timing. Today the two worlds are converging, as carriers turn their central offices into edge data centers and data center operators host carrier equipment. This lesson explains telecommunications facilities, the networks that connect homes and mobile phones, how DC plants are sized, and how networks keep time.

Central offices and edge sites

A central office (CO), or telephone exchange, is a carrier building that terminates the copper or fiber lines from the homes and businesses around it, typically within a few kilometers in a city, and connects them to the carrier's backbone. Large cities have dozens of COs, often in windowless buildings built decades ago with heavy floors, battery rooms and cable vaults.

Three trends are changing these buildings:

• Old copper switching and transmission equipment is being retired, freeing space and power.
• Fiber access and mobile networks need compact, distributed equipment close to users.
• Latency-sensitive services, such as mobile network functions, content caches and industrial applications, want computing within a few milliseconds of users.

As a result, carriers are converting COs into small edge data centers running standard servers, while data center operators build edge sites of their own. Edge sites face distinctive constraints: limited space and power (often tens to a few hundred kilowatts), existing buildings with limited cooling, little or no on-site staff, and the need to be managed remotely.

Access networks: how users connect

Fiber to the home

Most new fixed broadband uses passive optical networks (PONs). A single fiber from an optical line terminal (OLT) in the CO is split by unpowered optical splitters, typically 1:32 or 1:64, to reach many homes, each with an optical network terminal. Downstream, the OLT broadcasts to all homes on the split; upstream, the homes take turns in time slots. Generations include GPON (2.5 Gb/s down, 1.25 Gb/s up), XGS-PON (10 Gb/s symmetric) and the emerging 25G and 50G PON.

Mobile networks

A mobile network has three parts: the radio access network (RAN) of antennas and radios on towers and rooftops, the transport network that connects cell sites to the core (fronthaul from radios to baseband processing and backhaul onward), and the core network that handles authentication, mobility and connections to the internet. In 5G, the core runs as software on servers in data centers, and parts of the RAN baseband processing can be centralized in edge sites, which is a major driver of edge data center demand. Macro cell sites draw a few kilowatts each, supplied by small DC power plants with batteries in a cabinet or shelter at the base of the tower.

The −48 V DC power plant

Telecommunications equipment has traditionally run on −48 V DC rather than AC. The voltage is low enough to be considered relatively safe to touch (below about 60 V DC) yet high enough to keep currents manageable. The positive terminal is grounded and the supply is negative with respect to ground, because a negative potential reduces electrolytic corrosion of outside cables and connections. The key advantage is that batteries can float directly on the DC bus with no inverter between them and the load, making the power path very simple and reliable.

A DC plant consists of:

• Rectifiers: modular AC-to-DC converters, typically 2 kW to 4 kW each, hot-swappable and installed with N+1 or better redundancy.
• Batteries: strings of cells, traditionally 24 vented or valve-regulated lead-acid cells of 2 V, increasingly lithium-ion. They float at about 54 V (2.25 V per cell for lead-acid) and supply the load instantly if AC fails.
• Distribution: busbars, fuse panels and breakers feeding the equipment, often through battery distribution fuse bays (BDFBs) near the equipment rows.
• Controller: monitors voltages, currents, temperatures and alarms, and manages charging and low-voltage disconnects.

Because there are no inverters and the batteries are always connected, a DC plant's availability is very high. Its main weakness is that low voltage means high current: a 100 kW load at 48 V draws over 2,000 A, requiring large copper conductors, and voltage drop must be controlled carefully (ELEC-110, Lesson 11).

Data centers have adopted some DC ideas: open rack designs distribute 48 V DC or higher-voltage DC within racks, and some facilities use 380 V DC distribution. Carrier equipment installed in colocation facilities often still requires −48 V, so carrier-neutral data centers provide DC plants in their meet-me rooms and carrier spaces.

Telecom equipment standards

Equipment installed in North American carrier facilities is traditionally required to meet NEBS (Network Equipment-Building System) criteria, originally developed by Bellcore and later published by Telcordia, as GR-63-CORE (physical protection: temperature, humidity, altitude, earthquake, fire spread, airborne contaminants, acoustic noise) and GR-1089-CORE (electromagnetic compatibility, lightning and power-fault protection, and electrical safety). NEBS equipment must, for example, survive simulated earthquakes for the highest seismic zone and operate for short periods at temperatures up to 55 °C. Data center equipment is generally not NEBS-certified, which is one reason why carriers and data center operators sometimes struggle to share space.

In the US, the FCC regulates interstate telecommunications, manages spectrum with NTIA, and requires carriers to report significant outages through its Network Outage Reporting System (NORS). Equipment must be FCC-authorized, and carriers must meet requirements for emergency backup power at cell sites in some states.

Timing and synchronization

Networks need time in two senses. Frequency synchronization keeps every device's clock ticking at the same rate; phase or time synchronization keeps them showing the same time. Mobile networks need both: 5G networks using time-division duplexing, in which uplink and downlink share a channel in alternating slots, require neighboring cell sites to be aligned within about ±1.5 µs or they interfere. Financial trading systems must timestamp trades to microsecond accuracy for regulators, and distributed databases use synchronized clocks to order transactions.

Time is distributed from primary references in a hierarchy:

• GNSS (global navigation satellite systems, such as GPS) receivers provide time traceable to UTC within tens of nanoseconds, from antennas with a view of the sky.
• PTP (Precision Time Protocol, IEEE 1588) carries time across a packet network. A grandmaster clock, locked to GNSS, sends timestamped messages; switches along the path act as boundary or transparent clocks that correct for their own delays, achieving sub-microsecond accuracy across a network. Telecom profiles (ITU-T G.8275.1) define how to do this for mobile networks.
• Synchronous Ethernet (SyncE) distributes frequency through the physical layer of Ethernet links.
• NTP (Network Time Protocol) distributes time to ordinary servers to within about a millisecond over the internet, or much better on a local network.

PTP's calculation assumes the path delay is the same in both directions:

Any asymmetry, such as different fiber lengths in the two directions, appears directly as a time error of half the difference, so timing networks are engineered and calibrated carefully.

Holdover and GNSS vulnerability

GNSS signals are extremely weak at the earth's surface and can be lost to antenna faults, local interference, deliberate jamming or spoofing, which has become common near conflict zones and is sometimes caused by illegal personal jammers near highways. When the reference is lost, clocks enter holdover, running on their own oscillators, whose frequency error causes time to drift.

Satellites

Satellites reach places that fiber and cellular networks do not: ships, aircraft, remote communities and disaster areas. Geostationary satellites orbit at 35,786 km, so a signal traveling up and down for a request and again for the reply covers at least four times that distance: the minimum round-trip time is about 480 ms, and practical latency is 550 ms to 700 ms. Low-earth-orbit (LEO) constellations such as Starlink, Eutelsat OneWeb and Amazon's Project Kuiper orbit at a few hundred to about 1,200 km, giving round-trip times of roughly 25 ms to 60 ms in practice. LEO constellations connect to the internet through ground stations, which are often located at or near data centers and carrier hotels.