Leaf-spine and data centre fabrics

Lesson 1 followed a single frame down a single link. A data centre has tens of thousands of links, and how they are arranged decides how much traffic the building can carry, how many failures it can survive, how far the cables run and how much the network costs. That arrangement is the network topology, and the collection of switches and links that connects every server to every other server is called the fabric.

Almost every data centre built in the last decade uses the same basic topology, called leaf-spine. This chapter explains why the older three-tier design gave way to it, how to size a leaf-spine fabric with simple arithmetic, how it grows to tens of thousands of servers, and the protocols that run on top. It finishes with what the topology means for the building: where the network racks go, how many fibres are needed and how far they can run.

Traffic patterns: north-south and east-west

Network engineers describe traffic by direction on a diagram where the internet is at the top and the servers are at the bottom. North-south traffic flows between the servers and the outside world: a user's request coming in and the response going out. East-west traffic flows between servers inside the data centre.

In the early 2000s most traffic was north-south: a request arrived, one server answered it, and the answer left. Modern applications are different. A single web page may be assembled from dozens of microservices running on different servers, each calling databases, caches and storage systems on yet more servers. Big-data analytics shuffles data between hundreds of machines. Distributed storage keeps three copies of every block on different racks. AI training synchronizes every accelerator with every other after each step. In a typical modern data centre, east-west traffic is many times larger than north-south traffic, and in AI clusters it is almost all of it.

A topology designed for north-south traffic struggles when most traffic goes sideways. That is the story of the three-tier network.

The three-tier network

The classic enterprise data centre network had three layers of switches:

• Access switches at the top of each rack or end of each row, connecting servers.
• Aggregation (distribution) switches, each pair connecting a group of access switches.
• Core switches connecting the aggregation pairs to each other and to the internet.

Access and aggregation were usually joined at layer 2, with the Spanning Tree Protocol blocking redundant links to prevent loops, and routing happened at the aggregation or core. The design worked well for north-south traffic, but it had three problems as east-west traffic grew.

• Unequal paths: two servers on the same access switch were one hop apart, while two servers in different aggregation blocks had to climb to the core and back down, crossing five switches. Latency and available bandwidth depended on where a server happened to sit.
• Wasted links: Spanning Tree blocks redundant links, so half of the expensive uplinks often sat idle, carrying traffic only after a failure.
• Bottlenecks at the top: all traffic between aggregation blocks squeezed through a few large core switches, which were expensive, hard to scale and large failure domains. Losing a core switch could halve the capacity of the whole data centre.

Leaf-spine

A leaf-spine fabric has just two layers. Leaf switches connect to servers, usually one or two at the top of each rack. Spine switches connect only to leaves. The rule is simple:

• Every leaf connects to every spine.
• Leaves do not connect to each other, and spines do not connect to each other.
• Servers connect only to leaves.

As a result, any server can reach any server in another rack in exactly three switch hops: up to its leaf, up to a spine, and down to the destination leaf. Every pair of racks has the same latency and the same number of paths. With \(S\) spines, there are \(S\) equal-cost paths between any two leaves, and all of them carry traffic at once.

The idea is not new. It is a form of Clos network, described by Charles Clos at Bell Labs in 1953 for telephone switching, where many small switches arranged in stages could act like one huge switch. A leaf-spine fabric is a "folded" three-stage Clos network: the first and last stages are the same leaf switches, folded back on themselves. Because it is built from many identical, relatively inexpensive switches rather than a few giant ones, it scales out simply by adding more.

Equal-cost multipath (ECMP)

For all the spines to share the load, each leaf must spread traffic across its uplinks. It does this with equal-cost multipath (ECMP) routing. The leaf sees several routes of equal cost to the destination, one through each spine, and chooses among them for each flow by computing a hash of fields in the packet header, typically the source and destination IP addresses, the protocol and the source and destination port numbers. All packets of one flow produce the same hash and take the same path, so they arrive in order; different flows spread roughly evenly across the uplinks.

ECMP works well when there are many small flows. It works less well with a few very large flows, sometimes called elephant flows. Two large flows that happen to hash to the same uplink will share it while other uplinks sit idle. AI training traffic consists of exactly this kind of large, synchronized flow, which is one reason AI fabrics use additional load-balancing techniques (Lesson 11).

Failure behaviour

Because every leaf connects to every spine, losing a spine removes one path out of \(S\) everywhere, but nothing becomes unreachable. The fabric loses \(1/S\) of its capacity. With four spines, a spine failure cuts capacity by 25%; with sixteen, by about 6%. This "graceful degradation" is a major advantage: maintenance on a spine can be done during business hours by draining traffic away from it first. Designers choose the number of spines partly to keep the impact of one failure acceptable.

Losing a leaf takes down the servers attached only to that leaf. For that reason servers that must survive a leaf failure are dual-homed to two leaves in the same rack, and applications are spread across racks so that losing one rack is tolerable.

Sizing a leaf-spine fabric

Sizing a fabric is arithmetic with a few quantities:

• The radix of a switch: its number of ports at a given speed. A switch chip's total capacity is fixed (for example 51.2 Tb/s), so it can be configured as 64 ports of 800 Gb/s, 128 of 400 Gb/s or 256 of 200 Gb/s.
• Leaf downlinks (to servers) and uplinks (to spines).
• The number of spines, which equals the number of uplinks per leaf when each leaf has one link to each spine.
• The number of leaves, which can be at most the number of ports on each spine.

The maximum number of server ports in a two-layer fabric is therefore:

Oversubscription

A leaf's oversubscription ratio compares the bandwidth it offers to servers with the bandwidth it has towards the spines:

A ratio of 1:1 is called non-blocking: if every server sends at full speed to servers in other racks, the uplinks can carry it all. A ratio of 3:1 means the uplinks can carry only a third of the worst case. Oversubscription is a cost decision. Servers rarely all transmit at full rate at once, so general-purpose cloud and enterprise fabrics commonly run at 2:1 or 3:1 at the leaf. Storage-heavy and high-performance computing fabrics aim for lower ratios, and AI training back-end fabrics are almost always built non-blocking, because every accelerator really does transmit at full rate at the same moment.

Notice how the spine radix sets the size limit: to add a 65th leaf, a 64-port spine has no free port. The fabric must then either use bigger spines or add a third layer.

Growing beyond one fabric: five-stage Clos

When a two-layer fabric is not big enough, designers add a layer, creating a five-stage Clos network. The building is divided into pods (sometimes called fabric modules or clusters). Each pod is a leaf-spine fabric in its own right, but its spines use half their ports to connect down to the pod's leaves and half to connect up to a layer of super-spine switches. The super-spines are arranged in planes: spine number 1 in every pod connects to the switches in super-spine plane 1, spine number 2 to plane 2, and so on.

Traffic within a pod still takes three hops. Traffic between pods takes five: leaf, spine, super-spine, spine, leaf. The fabric can now grow by adding pods, up to the radix of the super-spines. Hyperscale operators publish designs of this kind that connect over 100,000 servers in a single building or campus.

Routing the fabric: layer 3 to the rack

Early leaf-spine designs stretched layer 2 VLANs across the fabric so virtual machines could move between racks without changing IP addresses. Large layer 2 domains bring the broadcast and loop problems described in Lesson 1, so modern fabrics route at layer 3 all the way down to the leaf. Each leaf is the default gateway for its own rack's subnets, and every link between a leaf and a spine is a small routed link.

Many large operators run BGP (Border Gateway Protocol), the same protocol that ties the internet together, as the routing protocol inside the fabric, following a design published as RFC 7938. Each leaf (or each group of leaves) gets its own private autonomous system number, and the switches tell each other which subnets they can reach. BGP is well understood, scales to enormous numbers of routes and gives operators fine control over traffic. Smaller fabrics often use OSPF or IS-IS instead. This routed network of switches and links is called the underlay.

Overlays: VXLAN and EVPN

Routing to the rack is simple and robust, but some workloads still need layer 2 connectivity between racks, and cloud providers need to give each customer an isolated private network that can span the whole data centre. The answer is an overlay: a virtual network built on top of the routed underlay by wrapping (tunnelling) one packet inside another.

The most common overlay is VXLAN (Virtual Extensible LAN). A VXLAN tunnel endpoint, which can be a leaf switch or software in the server's hypervisor, takes the original Ethernet frame and wraps it in a new UDP/IP packet addressed to the tunnel endpoint at the destination. The underlay simply routes that packet like any other. VXLAN identifies each virtual network with a 24-bit VXLAN network identifier, giving over 16 million segments compared with 4094 VLANs.

The outer headers add 50 bytes: 14 for the outer Ethernet header, 20 for IPv4, 8 for UDP and 8 for the VXLAN header. A tenant frame with a 1500-byte payload therefore becomes a 1550-byte packet in the underlay, which is why fabrics are built with a large MTU (commonly 9216 bytes on the switches) from the start.

Tunnel endpoints also need to know which MAC and IP addresses live behind which other endpoints. Rather than learning this by flooding, modern fabrics use EVPN (Ethernet VPN), an extension of BGP that distributes address information as routes. The combination, usually called EVPN-VXLAN, is the standard way to build multi-tenant layer 2 and layer 3 networks on a routed leaf-spine underlay. Lesson 4 looks at how tenants are isolated and secured on top of it.

Where the switches go

The topology has to be turned into racks, cables and pathways, and this is where network design meets facility design.

Top of rack, end of row and middle of row

• Top of rack (ToR): one or two leaf switches sit in each server rack. Servers connect with short DAC cables inside the rack, and only the uplinks leave the rack, on fibre. This is the most common arrangement in cloud and hyperscale facilities. It minimizes cabling between racks but means many switches, each a separate device to power, cool and manage.
• End of row (EoR) or middle of row (MoR): larger leaf switches sit in dedicated network racks at the end or middle of each row, and every server is cabled to them. This uses fewer, larger switches and keeps network gear in one place, at the cost of many more cables running along the row. It is common in enterprise data centres and some AI clusters, where back-end leaves for a group of GPU racks are placed together.

Spines, super-spines and the routers that connect to carriers sit in network rows, often in a central area so that cable runs to every hall stay short. In TIA-942 terms (covered in DCE-320), these spaces are the main distribution area and horizontal distribution areas.

Distance and reach

Every link type has a maximum length. Multimode fibre optics such as 100GBASE-SR4 and 400GBASE-SR8 reach about 100 m on OM4 fibre; single-mode optics such as 400GBASE-DR4 reach 500 m, and FR4 reaches 2 km. The route length counts, not the straight-line distance: cable climbs from the rack into an overhead tray, follows the tray around corners, drops into the network row and loops through patch panels. A rule of thumb is to add 20 to 30 percent to the plan distance plus several metres at each end. If the longest leaf-to-spine run in a large hall exceeds 100 m, the design must use single-mode optics for those links, which changes the fibre plant for the whole building. This is a decision to make early, together with the network architect.

The cable plant must also be sized for the number of links. In the 4096-server example above, the 1024 leaf-spine links each need a pair of fibres for a duplex optic, or eight fibres for a parallel optic like DR4. That is thousands of fibres converging on the network row, with trays, patch panels and space for growth. Lesson 9 and DCE-320 cover how this plant is designed.

Power and cooling for network equipment

Network switches are smaller loads than servers but are not trivial. A modern 51.2 Tb/s switch with 64 ports of 800 Gb/s draws roughly 1.5 to 3 kW including its optics, and a network rack full of them can reach 15 to 30 kW, comparable with a dense server rack. Many switches move air from front to back, but some move it back to front ("reverse airflow") to suit their position in the rack; the airflow direction must match the hot-aisle and cold-aisle layout, or the switch will draw in hot exhaust air. Network rows also need A and B power feeds like any other critical equipment, since a spine failure affects every rack in the fabric.