Network and connectivity at a glance

Power and cooling keep the servers alive, but a server that cannot talk to anything is useless. Connectivity is the third essential service of a data centre, and for many colocation customers it is the main reason they chose the building. In this lesson Jamal follows a fibre from the street to a customer's server, learns the names of the network equipment he will see every day and learns why technicians handle cabling with such care. The details of switching, routing and fibre work belong to later courses; here the goal is to know what is where and why.

Getting connected to the world

Data centres connect to the outside world through fibre optic cables owned by carriers: telephone companies, cable companies, long-distance fibre networks and specialist data centre network providers. The cables run in underground conduit along roads and rail lines and enter the building through the entrance rooms described in Lesson 4. A well-connected site has fibre from many carriers, arriving by different routes from different directions, so that a backhoe cutting one duct does not isolate it. This is called route diversity, and customers ask about it as carefully as they ask about power.

YUL-2 has two entrance rooms, one on the north side of the campus and one on the south, fed by eight carriers along two road corridors and one rail corridor. Carrier fibre reaches downtown Montréal's carrier hotels, the internet exchange point there and the long-haul routes to Toronto, Quebec City and New York.

The meet-me room and cross-connects

In a colocation building, carriers place their equipment in the meet-me room. A customer that wants to buy service from a carrier orders a cross-connect: the provider installs a fibre or copper cable from the carrier's panel in the meet-me room to the customer's panel in its cage. Once the cross-connect is in place and tested, the carrier turns up service over it. Cross-connects also link customers to each other, to cloud on-ramps and to internet exchange ports.

Each cross-connect has a circuit identifier and is recorded in the site's database with both ends: panel, position and port. Technicians install them to a work order, label both ends, test the light level or continuity and close the order with the results. Removing the wrong cross-connect can cut a customer off from a carrier, so removals are checked against the database and the labels at both ends before anything is unplugged. Lesson 11 shows how such a job runs from ticket to closure.

Network equipment in the racks

Inside each customer cage and each cloud hall, the network looks broadly the same. At the top of most racks is a patch panel where the structured cabling from the row terminates. Below or beside it sits a switch, often called a top-of-rack switch, that connects all the servers in that rack. The top-of-rack switches connect upward to larger switches that serve the row or the hall, often arranged in a design called leaf and spine: every leaf switch in the racks connects to every spine switch, so that any server can reach any other in two hops. At the edge of the network, routers connect it to the carriers and the internet. Firewalls and load balancers sit between the outside world and the servers.

• Patch panel: a passive panel of ports where permanent cables terminate and short patch cords make connections.
• Switch: an active device that connects servers and forwards traffic between them.
• Router: an active device that connects networks together and to the outside world.
• Transceiver: a small plug-in module that converts electrical signals to light for a fibre port; its type must match the fibre and the far end.
• Patch cord: a short cable with connectors at both ends, copper or fibre.

Many servers also have a separate management port, connected to a separate management network. Through it, customers and operators can reach the server's remote management controller, see its health, power it on and off and watch its console, even when the server's main network is down. Much of what remote hands requests ask technicians to do, such as checking a status light, is needed only when this management access has failed too.

Copper and fibre

Two kinds of media carry data in a data centre. Copper twisted-pair cable, the familiar Ethernet cable with RJ45 connectors, is used for short links of up to 100 m, mostly from servers to top-of-rack switches and for management networks. Fibre optic cable carries light through glass strands and is used for longer links and for the highest speeds. Data centres use both singlemode fibre, with a very small core, for long distances and most new high-speed links, and multimode fibre, with a larger core, for shorter links. Connectors include the LC, a small duplex connector, and the MPO, a multi-fibre connector holding 8, 12, 16 or 24 fibres in one ferrule, which is used for the high-speed links of AI clusters. The details of both belong to later courses on structured cabling and fibre.

Fibre is fragile in ways copper is not. A fibre bent too tightly loses light and can crack. A speck of dust on a connector end face can block enough light to cause errors or damage the connector it is mated to. The rules technicians learn first are to keep dust caps on unused connectors, to inspect and clean every fibre connector before mating it, and never to look into the end of a fibre or a transceiver, because some carry invisible laser light.

Cable management

A data hall carries tens of thousands of cables. Good cable management keeps them organized, labelled, supported and out of the airflow, so that anyone can find, trace and change a connection without disturbing the others. Poor cable management, sometimes called spaghetti, blocks airflow, hides labels, strains connectors and makes every change risky. The basic rules are:

• Every cable is labelled at both ends with an identifier that matches the records.
• Cables are routed in trays, managers and lacing bars, not draped across equipment or the fronts of other servers.
• Hook-and-loop straps hold bundles; tight plastic cable ties crush cables and are not used on data cabling.
• Each cable is the right length, with neat service loops, not coils of spare cable stuffed into the rack.
• Power and data cables are kept apart, usually on opposite sides of the rack.
• Nothing blocks the fronts or backs of servers where air enters and leaves.

Customers inspect their cages and judge a provider partly by the state of its cabling. Roz judges new technicians the same way.

How much room a network needs

Network equipment and cabling take space that cannot be used for servers. Every hall has racks set aside for network distribution, rows of patch panels and room for trays overhead. When planners decide how many server racks fit in a hall, they subtract the space for aisles, cooling units, network racks and clearances first. The example below shows how quickly the count falls from the naive figure.

# How many server racks fit in a hall?
hall_area = 1200          # usable hall floor, m2
rack_footprint = 0.72       # one 600 x 1200 mm cabinet, m2
gross_factor = 2.6          # aisles, clearances and access, typical for air-cooled rows
cooling_units = 12
cooling_area_each = 9  # m2 each including service space
network_racks = 24
naive = hall_area / rack_footprint
usable = hall_area - cooling_units * cooling_area_each
positions = usable / (rack_footprint * gross_factor)
server_racks = int(positions) - network_racks
print(f"Naive count (wall-to-wall racks): {naive:.0f}")
print(f"Floor left after cooling units  : {usable:,.0f} m2")
print(f"Rack positions after aisles     : {positions:.0f}")
print(f"Server racks after network racks: {server_racks}")

Naive count (wall-to-wall racks): 1667
Floor left after cooling units  : 1,092 m2
Rack positions after aisles     : 583
Server racks after network racks: 559

The naive count assumes racks could cover the whole floor. Once aisles, cooling units and network racks are counted, the hall holds about a third as many server racks. That is the hall area per rack figure from Lesson 4 seen from the other side.

Latency: why distance matters

Light in fibre travels at about two thirds of its speed in a vacuum, roughly 200,000 km a second, or 5 microseconds per kilometre. That sounds instantaneous, but for some applications every millisecond counts. Trading systems want to be as close as possible to an exchange; online games and video calls want to be close to their users; AI clusters want every processor as close as possible to every other. Inside a building, the cable length between servers in the same AI cluster is planned so that every link stays within the reach of its transceivers and within the timing the cluster expects. Between buildings and cities, latency is one reason data centres cluster in a few well-connected markets and edge sites exist at all.

# Round-trip time over glass for some distances (light in glass, about 5 microseconds per km)
routes = [("Hall D to Hall A, same campus", 1), ("Campus to downtown carrier hotel", 40), ("Campus to Toronto", 550), ("Campus to New York", 600), ("Campus to Vancouver", 4800)]
for name, dist in routes:
    km = dist * 1
    rtt_ms = 2 * km * 5 / 1000
    print(f"{name:34s} {dist:6,d} km  round trip about {rtt_ms:5.2f} ms")

Hall D to Hall A, same campus           1 km  round trip about  0.01 ms
Campus to downtown carrier hotel       40 km  round trip about  0.40 ms
Campus to Toronto                     550 km  round trip about  5.50 ms
Campus to New York                    600 km  round trip about  6.00 ms
Campus to Vancouver                 4,800 km  round trip about 48.00 ms

Real routes are longer than straight lines and add equipment delay, but the arithmetic shows why the physical route of a fibre matters to customers and why carriers advertise their latency.

The technician's network work

On the network side, a data centre technician's typical tasks include installing and labelling patch cords and cross-connects, cleaning and inspecting fibre connectors, testing links with light meters or cable testers, seating transceivers, reading port and status lights for a customer, tracing a cable to confirm where it goes, and moving connections when a customer asks. Configuring switches and routers is usually the customer's or a network engineer's job. A technician may connect a laptop to a console port at a customer's request, but only under a ticket and following the customer's instructions exactly.

Labels and records

Every connection in the building exists twice: once as a physical cable and once as a record. The record shows each end of the cable, the panel and port, the circuit or cross-connect identifier, the customer and the date it was installed. Labels on the cable and the panel carry the same identifier. When the two agree, any technician can find and change a connection safely. When they disagree, the record wins only after it has been checked, because a cable moved without updating the record is a trap for the next person. Technicians update the record in the same ticket as the physical work, never later.

Audits compare the records with the floor. A cage audit lists every cable leaving the cage and checks it against the cross-connect database. Cables that match no record are traced, as in this lesson's example, and either recorded or removed with the customer's agreement.