Networks and connectivity

A data center without network connections is just an expensive heater. Its value comes from moving data: between servers inside the building, to other data centers, and to the users and businesses it serves. Connectivity shapes where data centers are built, how their buildings are laid out, and how much they are worth: facilities where many networks meet command premium prices, and AI training clusters are limited by how fast their thousands of processors can exchange data. This lesson gives a facility engineer's view of networking: the basic units, the speed of light as a design constraint, how networks are arranged inside the building, how fiber enters and leaves it, and the spaces and pathways that cabling needs. DCE-210 and DCE-320 cover network and cabling design in depth.

Bits, bandwidth and latency

Network capacity (bandwidth) is measured in bits per second: a 100 Gb/s (gigabits per second) link carries 100 billion bits each second, or 12.5 gigabytes per second. Server connections have grown from 1 Gb/s two decades ago to 25, 100, 200 and 400 Gb/s today; switch-to-switch links run at 400 Gb/s and 800 Gb/s, with 1.6 Tb/s arriving. AI servers often have eight network connections of 400 Gb/s or 800 Gb/s each, one per GPU.

Latency is the delay for data to travel from one point to another. Much of it comes from the speed of light in optical fiber, which is about two thirds of its speed in a vacuum, roughly 200,000 km/s, or 5 microseconds per kilometer one way. Round-trip time (RTT) is twice the one-way delay plus processing in switches and routers.

Why location and connectivity go together

Latency and connectivity explain several patterns in where data centers are built:

• Interconnection hubs: certain cities and buildings host many carriers, cloud providers and internet exchanges, so new data centers cluster nearby to connect to them cheaply and with low latency.
• Availability zones: cloud providers build groups of data centers tens of kilometers apart, close enough for fast synchronous connections but far enough apart that a local disaster is unlikely to hit more than one.
• Edge data centers: small facilities near users serve applications that need very low latency, such as content delivery, gaming and some industrial uses.
• Remote training campuses: AI training is less sensitive to latency from users, so very large training campuses can be built where power and land are available, as long as high-capacity fiber can reach them.

In the US, major interconnection hubs include Northern Virginia (Ashburn), the New York area, Chicago, Dallas, Silicon Valley and Los Angeles, where large internet exchanges and carrier hotels concentrate networks.

Networks inside the building

Inside a data center, servers connect to switches, which connect to other switches in a hierarchy. Modern data centers mostly use a leaf-spine architecture: every rack (or pair of racks) has leaf switches at the top of the rack, and every leaf connects to every spine switch, so any two servers are separated by the same small number of hops. This suits the east-west traffic between servers that dominates modern applications. Larger facilities add a third layer (super-spines) to connect many leaf-spine pods.

AI clusters add a separate back-end network that connects GPUs in different servers to each other with very high bandwidth and low latency, typically using InfiniBand or specialized Ethernet. A cluster of tens of thousands of GPUs may need hundreds of thousands of optical connections. Because cable lengths are limited (especially for copper links of a few meters) and every meter adds latency, the network design constrains how racks are arranged and how large a single hall can be.

Structured cabling and spaces

Cabling in a data center is organized as structured cabling: permanent, tested links between defined spaces, with patch panels where connections are made and changed, rather than ad hoc cables run directly between devices. The ANSI/TIA-942 standard for data center infrastructure defines a hierarchy of spaces:

• Entrance room (ER): where carriers' outside cables enter the building and terminate; often two, at opposite sides of the building, for diversity.
• Main distribution area (MDA): the core of the network, with core routers and switches and the main cross-connects.
• Horizontal distribution area (HDA): intermediate distribution serving groups of rows.
• Equipment distribution area (EDA): the racks themselves.
• Zone distribution area (ZDA): an optional consolidation point in the floor or overhead.

Colocation facilities also have a meet-me room (MMR), a secure room where customers' cages connect to carriers and to each other through cross-connects, which the operator installs and charges for. Cross-connect revenue is an important part of the colocation business.

Copper and fiber

Short connections inside a rack often use copper: direct-attach copper (DAC) cables carry 100 Gb/s to 800 Gb/s over a few meters with very low power and cost. Longer links use optical fiber with transceivers that convert electrical signals to light:

• Multimode fiber, with a larger core, is used for shorter runs (up to about 100 m at high speeds) with lower-cost transceivers.
• Single-mode fiber, with a very small core, carries signals for kilometers and is used between halls, buildings and cities; it is increasingly used throughout new data centers.

High-density fiber uses multi-fiber MPO connectors carrying 8, 12, 16 or 24 fibers in one connector. Every connection and splice loses some light, so designers calculate a loss budget to make sure the light reaching the receiver is strong enough (DCE-320).

Pathways and physical design

Cables need pathways: overhead ladder racks and basket trays, fiber raceway systems, underfloor trays in raised-floor rooms, and conduits between rooms and buildings. Designers must:

• Size trays for the expected cable count with room to grow (fill is usually limited to about 50% of tray depth to allow additions and airflow).
• Coordinate trays with busways, lighting, sprinklers, containment and liquid cooling pipes, all competing for the same overhead space.
• Keep the minimum bend radius of fiber and protect it from damage.
• Separate the A and B network paths physically, as with power.
• Provide fire-stopping where cables pass through rated walls (Lesson 9).

Connecting the building to the world

Outside the building, fiber cables run to carriers' networks and other data centers along routes that should be physically diverse: two or more cables entering at different points, following different streets or rights of way, so that a single backhoe or construction accident cannot cut all connections. Fiber cuts from excavation are one of the most common causes of network outages. Large campuses build their own duct banks and lease or build dark fiber (unlit fiber that the customer lights with its own equipment) to other sites. Long-distance capacity is multiplied with dense wavelength division multiplexing (DWDM), which carries dozens of wavelengths, each at hundreds of gigabits per second, on one fiber pair.

Connectivity is planned with the site from the beginning (Lesson 11): how many carriers are nearby, how long new fiber routes would take to permit and build, and whether diverse routes are available. A site with ample power but no fiber may take years to connect.

Keeping the network running

Network equipment has its own reliability design. Servers usually connect to two leaf switches, so a switch can fail or be upgraded without disconnecting them; every leaf connects to several spines; and routing protocols redirect traffic within milliseconds when a link or switch fails. Power for network equipment follows the same A and B feed principles as servers, and the core network rooms (MDA and entrance rooms) are often placed in the most protected part of the building with their own cooling.

Most network outages are caused not by hardware but by configuration changes, software bugs and fiber cuts. Network teams therefore use automation to apply changes consistently, test changes in staging environments, roll them out gradually, and keep detailed documentation of every cable and port. Physical labeling matters: a single mislabeled patch cord can take down a customer, and audits of cable records against reality are a routine part of operations.