Carriers, peering and interconnection

A data center is only useful if it is connected. Customers need to reach the applications inside, data centers need to reach each other, and companies need fast private links to cloud providers. All of this travels over fiber owned by telecommunications carriers and over the agreements networks make with each other. For some facilities, the connections are the main product: a carrier-neutral colocation building in a major city can be worth more for the hundreds of networks that meet inside it than for its power and cooling.

This chapter explains how the internet is stitched together from independent networks, how those networks exchange traffic, how a data center is physically connected to carriers, what services carriers sell and how they are priced, and how engineers make connectivity resilient. It uses American examples: the carriers, exchange points and regulators a designer works with in the United States.

A network of networks

The internet is not one network. It is more than 70,000 independent networks, each called an autonomous system (AS), that have agreed to exchange traffic. An autonomous system is a network run by one organization with one routing policy: an internet service provider, a cloud provider, a university, a content company or a large enterprise. Each has a unique AS number. The networks tell each other which blocks of IP addresses they can reach using BGP (Border Gateway Protocol), the same protocol Lesson 3 described inside fabrics. When you load a web page, your request may cross three or four autonomous systems, each passing it on according to its own business relationships.

Those relationships come in two basic kinds.

Transit

In a transit relationship, a customer network pays a provider to carry its traffic to and from the entire internet. The provider announces the customer's addresses to everyone it connects with and gives the customer routes to everywhere. Small and medium networks, enterprises and many data center operators buy transit, usually from two or more providers for resilience.

At the top of the hierarchy are a few Tier 1 networks, such as Lumen, AT&T, NTT and Arelion, that can reach every destination on the internet without buying transit from anyone, because they peer with each other. Below them, Tier 2 networks both buy some transit and peer widely.

Peering

In a peering relationship, two networks exchange traffic directly, but only traffic between their own customers. Neither pays the other to reach the rest of the internet. Peering is often settlement-free: neither side pays, because both benefit roughly equally. It can also be paid, when one side benefits more. Peering reduces transit costs, shortens paths and lowers latency, because traffic goes directly from one network to the other instead of through a third party.

Content and cloud companies peer very widely. A large share of all internet traffic now flows directly from a few huge content and cloud networks into access networks, and a data center that hosts both kinds of network becomes a natural place for them to meet.

Where networks meet

Networks interconnect physically in a few ways.

• Internet exchange points (IXPs): an IXP operates a shared switching fabric, often spread over several data centers in a city. Each member network connects one or more ports to it and can then peer with any other member over that single connection. An IXP lets a network peer with hundreds of others without running hundreds of separate cables.
• Private network interconnects (PNIs): when two networks exchange a lot of traffic, they connect directly with dedicated links, usually cross-connects within the same building. PNIs carry the largest flows between big content providers and access networks.
• Carrier hotels: certain buildings in major cities, often older downtown office towers converted decades ago, became the places where long-haul fiber routes converged. Because every network was there, every new network came too. These buildings are extremely dense with interconnection and are very hard to replicate.

In the United States, major exchanges include DE-CIX New York, the Equinix Internet Exchange in Ashburn, Chicago, Silicon Valley and other metros, NYIIX in New York, Any2Exchange in Los Angeles and the Seattle Internet Exchange (SIX). Famous carrier hotels include 60 Hudson Street and 111 Eighth Avenue in New York, One Wilshire in Los Angeles, 350 East Cermak Road in Chicago and the Westin Building Exchange in Seattle. Northern Virginia, especially Ashburn in Loudoun County, grew from early interconnection points into the largest data center market in the world, in part because so many networks already met there.

Inside the building: the meet-me room

Inside a colocation data center, carriers bring their fiber into one or more meet-me rooms (MMRs). Each carrier terminates its fiber on its own patch panels or in its own small racks in the MMR. Each customer's cage or suite also has fiber running back to the MMR. To connect a customer to a carrier, or two customers to each other, the data center operator installs a cross-connect: a fiber patch cord, or a dedicated fiber run, between the two parties' panels.

Cross-connects are a recurring revenue stream for colocation operators, typically billed monthly per connection, and in a busy facility there may be tens of thousands of them. The operator keeps careful records of each one, because a single mislabeled fiber can take a customer offline. MMRs are high-security spaces with their own access control, separate from customer areas, and they need power, cooling and pathways like any other technical room. Large campuses usually have two MMRs on opposite sides of the building, each fed from a different entrance.

Carrier-neutral and carrier-specific facilities

A carrier-neutral data center lets any carrier bring fiber into the building and lets customers buy from any of them. Competition between carriers keeps prices down and gives customers choice and resilience. Most large colocation providers are carrier-neutral, and the number of carriers "on net" in a building is one of the main things customers compare.

A carrier-owned data center is run by a telecommunications company, and customers may be expected to use that carrier's services. Enterprise and hyperscale self-built data centers usually contract with a few carriers to bring diverse fiber to the campus and may also lay their own dark fiber to nearby colocation sites and to their other data centers.

What carriers sell

Data center operators and their customers buy connectivity in several forms, from raw glass to finished internet service:

• Dark fiber: unlit fiber strands, leased (often for 10 to 20 years under an indefeasible right of use, or IRU) between two points. The customer installs its own optical equipment and can run as much capacity as that equipment allows. Hyperscale operators use dark fiber between their data centers in a metro area.
• Wavelengths: a dedicated optical channel on the carrier's DWDM system (Lesson 9), typically 10, 100 or 400 Gb/s, between two locations. The carrier operates the optics; the customer gets a transparent pipe.
• Ethernet private lines and Ethernet virtual circuits: layer 2 connections between sites, sold at a committed bandwidth.
• IP transit (dedicated internet access): a port onto the carrier's network with full internet connectivity, priced by bandwidth.
• MPLS VPNs and SD-WAN: private wide-area networks connecting an enterprise's offices and data centers.
• Cloud on-ramps: private connections into public cloud providers, such as AWS Direct Connect, Microsoft Azure ExpressRoute, Google Cloud Interconnect and Oracle FastConnect. Enterprises use them to reach the cloud with predictable performance instead of over the public internet. Colocation facilities that host on-ramps attract enterprise customers for that reason.

Burstable billing and the 95th percentile

IP transit is often billed on the 95th percentile of usage. The carrier measures the traffic on the port every five minutes for the whole month, sorts the samples, discards the highest 5% and bills at the highest remaining value. In a 30-day month there are \(30 \times 288 = 8640\) five-minute samples; the top 432 (36 hours' worth) are discarded.

The method lets customers burst briefly above their usual traffic without paying for it, while still charging for sustained use. Usually there is also a commit: a minimum amount billed every month whatever the usage. Traffic is measured in each direction and the higher of the two 95th percentiles is normally used.

Connecting the building

Getting fiber into a data center is a construction project in its own right, and it is often on the critical path of the whole development.

Outside plant and entrances

Carriers reach a site through outside plant: fiber cables in underground conduits (ducts), on utility poles, or in trenches along roads and railways. At the property line, cables enter maintenance holes (manholes) or handholes, then run through the owner's conduits into a building entrance room. Good practice is at least two entrance points on different sides of the building, each fed by a separate conduit path from a separate street, so that a single excavator cannot cut every connection. The entrance conduits are usually a bank of several 100 mm (4 in) ducts, with spare ducts for future carriers.

In the United States, interstate telecommunications is regulated by the FCC (Federal Communications Commission), and state public utility commissions regulate some intrastate services. Carriers building on public streets need right-of-way permits or franchise agreements from cities and counties, and federal law gives them access to utility poles on regulated terms. On private land the data center owner grants easements. Major carriers include AT&T, Verizon, Lumen and Zayo, along with cable operators such as Comcast and Charter and many regional fiber providers; a designer typically asks several to quote for diverse routes to the site early in planning.

Bringing new fiber to a greenfield site can take a year or more, especially if it involves permits for road crossings, railways, rivers or highways. It is one of the reasons site selection (DCE-410) looks closely at existing fiber routes.

Route diversity and shared risk

Two connections are only independent if they share nothing that can fail. Experienced designers check for shared risk: places where two "diverse" routes actually run in the same conduit, cross the same bridge, pass through the same carrier building or follow the same railway corridor. Two carriers may even lease fiber in the same cable from a third party. Carriers can provide route maps (often under a non-disclosure agreement) so that the owner can check physical separation.

The same reasoning (series and parallel availability, from DCE-200) applies to entrance rooms, meet-me rooms, and the routers and power feeding them. A building with two carriers that both terminate in the same meet-me room on the same power feed has a single point of failure in the middle.

Latency and distance

Light in optical fiber travels at about two-thirds of its speed in a vacuum: roughly 204,000 km/s, which is about 4.9 µs per kilometer. Fiber seldom follows a straight line; it follows roads and railways, so the route is often 1.2 to 1.5 times the straight-line distance. Every millisecond matters to some applications: financial trading, online gaming, real-time collaboration and synchronous storage replication.

This physical limit shapes how cloud providers lay out their regions. Data centers in one availability zone, or a few zones in one region, are kept within roughly 100 km of each other so that synchronous replication stays within about a millisecond of round-trip time, while being far enough apart that a single local disaster is unlikely to hit all of them.

Long-distance and international traffic also travels on subsea cables that come ashore at cable landing stations. On the US East Coast, Virginia Beach has become an important landing point, with transatlantic cables such as MAREA and Dunant connecting back to the data centers of Northern Virginia.

Interconnection as a design driver

For a colocation operator, connectivity shapes the building: how many meet-me rooms and where, how many entrance conduits, how much space is reserved for carrier racks, and how fiber pathways run from the MMRs to every customer suite. For a hyperscale campus, it shapes the site: diverse fiber routes to the campus, dark fiber to other campuses in the region, and enough duct capacity for decades of growth. For an AI training campus, outside connectivity may be modest compared with the enormous internal fabric, but it still needs diverse routes for data ingestion, management and inference traffic.

In every case, the engineer's job is to make sure that the physical paths, rooms, power and security support the network relationships the business depends on, and that no single conduit, room or power feed can sever them.