Network hardware: switches, routers and transceivers

The fabrics of Lesson 3 are built from physical boxes that occupy rack space, draw power, produce heat and fail. This lesson opens those boxes: what is inside a modern data centre switch, how switch chips have grown, why the radix of a switch decides how large a fabric can be, how routers differ from switches, what the pluggable transceivers do and how much power they use, and the practical details that affect the facility, from airflow direction to spares.

Inside a switch

A data centre switch is built around a switching ASIC (application-specific integrated circuit), a single chip that receives, looks up, buffers and forwards every packet. Around it sit:

• SerDes (serializer/deserializer) circuits on the chip that drive the electrical lanes to the front-panel ports, today at 100 Gb/s or 200 Gb/s per lane.
• A packet buffer, usually on the chip and shared among all ports.
• Forwarding tables: MAC address tables, IP route tables and access control lists held in fast on-chip memory.
• A control-plane CPU, a small server on its own board running the network operating system, which runs routing protocols such as BGP, programs the ASIC's tables and provides management.
• Front-panel cages for pluggable transceivers, fans, and two or more hot-swappable power supplies.

The split between the data plane (the ASIC, forwarding packets at line rate) and the control plane (the CPU, deciding where packets should go) is fundamental. The CPU never touches ordinary packets; if it crashes, the ASIC may keep forwarding on its last instructions for a while, and if it is overloaded, routing protocols can time out even though the data plane is healthy.

Network operating systems

Traditionally each vendor supplied its switches with its own closed operating system. Large operators wanted to run the same software across hardware from several vendors and to change it themselves, which led to disaggregation: "white-box" or "bare-metal" switches sold without software, running an independent network operating system. SONiC, open-sourced by Microsoft in 2016 and now hosted by the Linux Foundation, is the most widely deployed open network operating system and runs on switches from many manufacturers. Vendors' own systems remain common in enterprises, where integrated support matters more than flexibility.

How switch chips have grown

Switch ASIC capacity has roughly doubled every two years, driven by faster SerDes lanes and more of them on each chip. Capacities of 12.8 Tb/s around 2018 grew to 25.6 Tb/s around 2020 and 51.2 Tb/s around 2023, and the first 102.4 Tb/s chip, Broadcom's Tomahawk 6, began shipping in mid-2025, supporting configurations such as 64 ports of 1.6 Tb/s or 512 ports of 200 Gb/s.

Radix: why port count matters

The same chip can be configured as fewer fast ports or more slower ports. A 51.2 Tb/s chip might provide 64 × 800 Gb/s, 128 × 400 Gb/s or 256 × 200 Gb/s, using breakout cables or optics that split one physical port into several logical ones. The number of ports is the switch radix, and it decides how large a fabric can be built in a given number of layers.

In a non-blocking two-layer leaf-spine built from identical switches of radix \(k\), each leaf uses half its ports for servers and half for uplinks, and each spine connects to up to \(k\) leaves. The maximum number of server ports is

and a three-layer fat tree (Lesson 11) reaches \(k^{3}/4\).

Switches and routers

Data centre switches and routers use similar hardware but are optimized differently:

• Data centre switches have very high capacity, low latency and relatively small tables and buffers, because inside a fabric there are only thousands of routes and the RTT is short.
• Routers at the edge of the data centre and in carrier backbones must hold the full internet routing table, roughly a million IPv4 routes and over 200,000 IPv6 routes in 2025, and growing. They need deep buffers (often gigabytes of external high-bandwidth memory) to handle long-RTT traffic, rich QoS, and features such as MPLS and large-scale encryption.
• Modular chassis routers and switches hold line cards and fabric cards in a single large enclosure, sometimes over 20U and 10 kW or more. They were once standard at the core of networks; in data centres they have largely given way to fabrics of fixed-configuration "pizza box" switches of 1U to 4U, which are cheaper, simpler and fail in smaller pieces.

Transceivers and cables

Each port takes a pluggable module whose form factor is named by the cage it fits:

• SFP and SFP28 for single-lane links up to 25 Gb/s (SFP56 and SFP112 for 50 and 100 Gb/s).
• QSFP28 and QSFP56 for four-lane 100 and 200 Gb/s.
• QSFP-DD and OSFP, with eight lanes, for 400 Gb/s and 800 Gb/s; OSFP is also used for 1.6 Tb/s.

The module may be an optical transceiver (Lesson 9), or the port may take a cable with the connectors built on: a direct-attach copper cable (DAC) for links of up to about 2 to 3 m within a rack, an active electrical cable (AEC) with signal-conditioning chips for a few metres more, or an active optical cable (AOC) with optics sealed into each end for longer runs. DACs use almost no power and are cheapest, so servers are connected to top-of-rack switches with copper wherever distance allows.

Optics power

As switch capacity has grown, the power used by the optics has grown with it, until in a fully populated switch the optics can use as much power as the switch itself.

Hardware details that matter to the facility

Airflow direction

Servers draw cool air in at the front and exhaust it at the rear. Switches, however, are often mounted with their ports facing the rear of the rack, where the server network cables are, so a switch's port side may face the hot aisle. Switches therefore come in two airflow versions, usually called port-side intake and port-side exhaust (vendors also use "front-to-back" and "back-to-front", whose meaning depends on which side the vendor calls the front). A switch mounted ports-to-the-rear in a hot-aisle/cold-aisle layout must use port-side exhaust, so that it takes air from the cold aisle through its power-supply side.

Getting this wrong is one of the most common installation errors in data centres: the switch ingests hot air, runs hot and may shut down, and its exhaust raises the inlet temperature of nearby servers. Fans and power supplies must all match; mixing directions in one switch is also a fault.

Power supplies and feeds

Switches have two or more hot-swappable power supplies, which should be fed from separate A and B power paths (DCE-200, Lesson 3). Supplies are available for AC input (100 V to 277 V) and for DC input, including −48 V DC for telecommunications facilities (Lesson 10) and higher-voltage DC in some data centres. A large switch's power supplies may each be rated 2 kW to 3 kW, and rack power plans should use the measured or vendor-typical load rather than the nameplate rating, which assumes worst-case optics and fan speeds.

Spares and failure rates

With thousands of switches and tens of thousands of transceivers, failures are routine events. Operators keep spares on site and design the fabric so that losing any one device causes only a small loss of capacity.

Lifecycle

Network equipment is typically refreshed every five to seven years, faster in AI clusters where each new generation of accelerators brings faster network interfaces. Each refresh changes rack power, heat and cabling. Data centre designs should leave room for the next generation of switches and the cable types it will need, rather than only today's.