Network layers and Ethernet

A data centre without a network is a warehouse of expensive heaters. Every server depends on the network to receive work, reach storage, talk to the other servers in its cluster and send answers back to users. In a modern AI cluster, the network can cost as much as 10 to 20 percent of the computing equipment, and a poorly designed one can leave thousands of accelerators idle while they wait for data.

Data centre engineers do not have to configure routers, but they do have to design the spaces, power, cooling, cable pathways and fibre plant that the network needs, and they work alongside network architects every day. To do that well, an engineer needs to speak the language: what the layers are, what Ethernet actually sends down the wire, how fast the links run and why those speeds come in the steps they do. This chapter builds that foundation. The rest of the course climbs from the single link to the fabric inside a building, the connections between buildings and carriers, the optical systems that carry the traffic, and the special networks built for AI training.

Why networks are built in layers

Sending data between two computers involves many separate problems: turning bits into light or voltage, deciding which device on a cable may talk, finding a path across many networks, recovering lost packets, and presenting the data to an application. Engineers handle this by dividing the work into layers. Each layer does one job, relies on the layer below and offers a service to the layer above. A web browser does not need to know whether its data crosses copper, fibre or radio; a fibre transceiver does not need to know whether it is carrying a video or a bank transfer.

Two layer models are in common use. The OSI (Open Systems Interconnection) reference model has seven layers and is mostly used as a vocabulary. The TCP/IP model, which describes how the internet actually works, has four or five. Engineers mix the two: they say "layer 2" and "layer 3" from the OSI model while using TCP/IP protocols.

The layers that matter most in a data centre

• Layer 1, physical: the cables, connectors, transceivers and the electrical or optical signals. This is where data centre engineers spend most of their time: fibre types, loss budgets, cable trays, patch panels and the power and heat of optics.
• Layer 2, data link: Ethernet. It moves frames between devices on the same network segment, using 48-bit MAC (media access control) addresses burned into each network interface. Switches work at this layer.
• Layer 3, network: IP (Internet Protocol). It moves packets between different networks using IP addresses, and routers choose the path. Most modern data centre fabrics route at layer 3 right down to the top-of-rack switch.
• Layer 4, transport: TCP (Transmission Control Protocol) provides a reliable, ordered byte stream and resends lost data; UDP (User Datagram Protocol) sends individual datagrams with no guarantees and is used where speed matters more than reliability, or where a higher layer handles recovery itself. Port numbers at this layer identify the application (443 for secure web traffic, for example). Lesson 2 looks at how TCP paces traffic and what limits its speed.
• Layers 5 to 7: application protocols such as HTTP for web traffic, DNS for name lookups, and storage protocols such as NVMe over Fabrics and iSCSI.

Encapsulation

As data passes down the layers, each layer adds its own header, a little like putting a letter in an envelope, then putting that envelope in a courier bag, then putting the bag in a shipping container. An application's data gets a TCP header (20 bytes without options), then an IP header (20 bytes for IPv4, 40 bytes for IPv6), then an Ethernet header (14 bytes) and a 4-byte trailer called the frame check sequence (FCS). At the other end each layer strips off its header and passes the contents up.

Every header is overhead: it uses link capacity but carries no user data. Engineers care about overhead because it decides how much of a link's headline speed is actually usable, which you will calculate shortly.

Ethernet

Ethernet was invented at Xerox PARC in 1973 to connect computers over a shared coaxial cable at about 3 Mb/s. Fifty years later the same name covers links running at 800 Gb/s over fibre, and the frame format is almost unchanged. Ethernet is standardized by the IEEE 802.3 working group, and each speed and medium has its own designation, such as 100GBASE-SR4: 100 Gb/s, baseband signalling, short reach, four lanes.

Early Ethernet devices shared one cable and had to take turns, detecting collisions when two talked at once. Modern data centre Ethernet is entirely switched and full duplex: each device has its own dedicated link to a switch port, both directions carry traffic at the full rate simultaneously, and collisions do not happen. A "100 Gb/s link" therefore carries 100 Gb/s in each direction at once.

The Ethernet frame

An Ethernet frame carries a payload of 46 to 1500 bytes in its standard form. The payload is usually an IP packet. Around it sit:

• Preamble and start frame delimiter (8 bytes): a fixed pattern that lets the receiver lock on to the signal. Not counted as part of the frame, but it uses time on the wire.
• Destination and source MAC addresses (6 bytes each).
• EtherType (2 bytes): says what the payload is, for example 0x0800 for IPv4 or 0x86DD for IPv6.
• Payload (46 to 1500 bytes). Short payloads are padded up to 46 bytes so the frame is at least 64 bytes long.
• Frame check sequence (4 bytes): a CRC-32 checksum. A receiver that finds a mismatch drops the frame.
• Interframe gap (12 bytes of idle time): a mandatory quiet period between frames.

The largest payload a link carries is its MTU (maximum transmission unit). The standard Ethernet MTU is 1500 bytes. Many data centres enable jumbo frames, usually with a 9000-byte MTU, so that storage and high-performance traffic spends less of the link on headers and the hosts handle fewer packets per second. Jumbo frames must be enabled consistently on every device in a path; a single device with a 1500-byte MTU in the middle will drop or fragment the large frames, and this is a classic cause of mysterious failures.

The bytes on the wire for one full-size frame are:

so even before IP and TCP are counted, Ethernet itself uses 38 of every 1538 bytes.

Small frames and packets per second

Overhead is worst for small frames. A minimum 64-byte frame still needs its 8-byte preamble and 12-byte gap, so it occupies 84 bytes, or 672 bits, of wire time. The maximum number of frames a link can carry per second is therefore:

For 10 Gb/s this is 14.88 million frames per second; for 100 Gb/s, 148.8 million. Switches and network cards are specified by whether they can forward at "line rate" for minimum-size frames, because handling each frame takes a fixed amount of work regardless of its size. Workloads with many small messages, such as some databases and AI inference serving, stress packets per second rather than bandwidth.

Serialization delay

Putting a frame onto a link takes time: the number of bits divided by the link rate. This is the serialization delay. A full-size frame of 1538 bytes (12,304 bits) takes 12.3 µs at 1 Gb/s, 1.23 µs at 10 Gb/s and 0.123 µs at 100 Gb/s. A switch that stores a whole frame before forwarding it adds at least one serialization delay at each hop, which is one reason faster links also reduce latency.

Switching and MAC addresses

A MAC address is 48 bits, written as six pairs of hexadecimal digits such as 3c:fd:fe:a4:01:7b. The first 24 bits identify the manufacturer (the organizationally unique identifier); the rest are assigned by the manufacturer. A layer 2 switch learns which MAC addresses live on which port by reading the source address of every frame it receives, and stores them in a MAC address table. When a frame arrives for a destination it knows, it sends the frame out of that one port only. When the destination is unknown, or the frame is a broadcast, it floods the frame out of every port in the same network segment.

Flooding is fine in a small network but becomes a problem as layer 2 domains grow: every broadcast reaches every host, MAC tables fill up, and a loop in the cabling can make frames circulate forever until the network collapses in a broadcast storm. The Spanning Tree Protocol prevents loops by blocking redundant links, but blocked links waste capacity. This is one of the main reasons modern data centres keep layer 2 domains small and route at layer 3, as the next chapter explains.

VLANs

A VLAN (virtual LAN) divides one physical switch network into separate logical networks. The IEEE 802.1Q standard inserts a 4-byte tag after the source MAC address containing a 12-bit VLAN identifier. Twelve bits give 4096 values, of which 0 and 4095 are reserved, so a network can have at most 4094 VLANs. That was plenty for an office building but far too few for a cloud provider with millions of customers, which is why data centres use overlay technologies such as VXLAN with a 24-bit identifier (over 16 million segments), covered in the next chapter.

IP addressing in brief

An IPv4 address is 32 bits, written as four decimal numbers such as 10.20.4.17. Addresses are grouped into subnets described by a prefix length: 10.20.4.0/24 means the first 24 bits identify the network and the last 8 bits identify hosts, giving 256 addresses. The first (network) and last (broadcast) addresses are reserved, leaving 254 usable host addresses. In general a subnet with prefix length \(p\) has \(2^{32-p}\) addresses.

The roughly 4.3 billion IPv4 addresses ran out years ago. Data centres use private address ranges such as 10.0.0.0/8 internally and translate to public addresses at the edge. IPv6 uses 128-bit addresses, written in hexadecimal such as 2001:db8:4a::17, and a standard subnet is a /64, which holds \(2^{64}\) addresses, more than enough for any rack. Large operators increasingly run IPv6 internally for exactly that reason.

Within a subnet, a host finds the MAC address that matches an IP address using ARP (Address Resolution Protocol) in IPv4 or Neighbour Discovery in IPv6. To reach anything outside its subnet, a host sends the frame to its default gateway, which in a modern data centre is usually the top-of-rack switch.

Ethernet speeds and lanes

Ethernet speed has grown by about a factor of ten every decade or so, and data centres have adopted each step as soon as it became affordable. The common data centre speeds today are 25 and 100 Gb/s to servers, 100 and 400 Gb/s between switches, and 400 and 800 Gb/s in AI clusters.

High speeds are built from lanes. A lane is one electrical or optical channel, and a switch chip connects to its ports through SerDes (serializer/deserializer) circuits, each running one lane. A 100 Gb/s port might be four 25 Gb/s lanes; an 800 Gb/s port is eight 100 Gb/s lanes. The industry moves forward by doubling the lane rate (10, 25, 50, 100 and now 200 Gb/s per lane), and each doubling allows a new top port speed with the same number of lanes. Ethernet at 1.6 Tb/s built on 200 Gb/s lanes is being standardized, and first products are already appearing.

Because lanes can be split, one physical port can often be broken out into several slower ports. A 400 Gb/s port with four 100 Gb/s lanes can serve four separate 100 Gb/s servers using a breakout cable. Breakouts appear constantly in cabling designs and affect how many fibres and connectors each rack needs.

NRZ and PAM4

Up to 25 Gb/s per lane, Ethernet used NRZ (non-return-to-zero) signalling: two signal levels, one bit per symbol. To go faster without doubling the symbol rate, 50 and 100 Gb/s lanes use PAM4 (four-level pulse amplitude modulation): four levels carry two bits per symbol. A 100 Gb/s PAM4 lane runs at about 53 gigabaud rather than 106.

PAM4 has a cost. With four levels squeezed into the same signal swing, the gap between levels is one third as large, so the signal is much more sensitive to noise. PAM4 links therefore rely on forward error correction (FEC): extra bits that let the receiver correct errors without retransmission. FEC adds a little latency (on the order of 100 nanoseconds) and some power. This is why the error-rate and loss specifications for modern optics and cables are so strict, and why a dirty connector that a 10 Gb/s link would tolerate can take down a 400 Gb/s link.

Bits and bytes

Network speeds are always quoted in bits per second; file and storage sizes are quoted in bytes. One byte is 8 bits. Mixing them up is the single most common error in network calculations, and it is an error by a factor of eight. Network rates also use decimal prefixes (1 Gb/s is \(10^9\) bits per second), while operating systems sometimes report memory and file sizes in binary units (1 GiB is \(2^{30}\) bytes, about 7% more than \(10^9\)). For quick engineering estimates, decimal units are fine; just say which you are using.

Copper or fibre?

At these speeds, signals on copper fade quickly. Inside a rack, the cheapest connection is a direct attach copper (DAC) cable: a fixed twinax cable with connectors on both ends and no optics, good for about 2 to 3 m at 100 Gb/s per lane, and drawing almost no power. Active electrical cables (AEC) add signal-conditioning chips to reach about 5 to 7 m. Anything longer uses fibre and optical transceivers, which convert electrical lanes to light and back. Transceivers plug into switch and network-card ports in standard form factors: SFP28 for 25 Gb/s, QSFP28 for 100 Gb/s, and QSFP-DD or OSFP for 400 and 800 Gb/s. Twisted-pair copper (category 6A cabling with RJ45 connectors) remains common for management networks, building systems and wireless access points at 1 to 10 Gb/s.

Optics matter to the facility engineer for three reasons. They are a large share of network cost. They consume power: an 800 Gb/s transceiver draws roughly 13 to 18 W, and a large AI cluster can have tens of thousands of them. And they decide which fibre type the building must provide and how much loss the cabling may have. Lesson 9 covers fibre and optics in depth.

The networks in a data centre

A data centre usually runs several separate networks, and each needs its own cabling, space and power:

• The production or front-end network carries traffic between servers and to the outside world.
• The storage network connects servers to storage systems; in many facilities it now shares the production Ethernet fabric (Lesson 5).
• The back-end or scale-out network, in AI and high-performance computing clusters, connects accelerators to each other for training. It is often the largest and most expensive network in the building (Lesson 11).
• The out-of-band management network connects each server's baseboard management controller (BMC), switch management ports and power distribution units, so that equipment can be controlled even when the production network is down. It runs at 1 Gb/s on copper and must be completely independent of the production network.
• Building and facility networks connect the building management system, electrical power monitoring, security cameras and access control. These operational technology networks are kept separate from IT networks for security.

Each of these networks shows up in the facility design as cable trays, patch panels, rack positions, switch power and cooling. Forgetting one, usually management or facility networks, is a common source of late changes during construction.

How this course is organized

The course moves from inside the rack to the wider world:

• Lesson 2 explains latency, TCP and quality of service: why networks feel fast or slow.
• Lessons 3 to 6 cover the data centre network: leaf-spine fabrics, virtual networks and security, storage networks, and the switches, routers and transceivers they are built from.
• Lessons 7 and 8 leave the building: carriers, peering and interconnection, then internet routing, content delivery and enterprise WANs.
• Lesson 9 covers optical transport, and Lesson 10 telecommunications facilities, −48 V DC power and timing.
• Lesson 11 examines the networks of AI clusters, and Lesson 12 network operations, reliability and the technologies ahead.