Carriers, peering and interconnection

A data centre is only useful if it is connected. Customers need to reach the applications inside, data centres need to reach each other, and companies need fast private links to cloud providers. All of this travels over fibre owned by telecommunications carriers and over the agreements networks make with each other. For some facilities, the connections are the main product: a carrier-neutral colocation building in a major city can be worth more for the hundreds of networks that meet inside it than for its power and cooling.

This chapter explains how the internet is stitched together from independent networks, how those networks exchange traffic, how a data centre is physically connected to carriers, what services carriers sell and how they are priced, and how engineers make connectivity resilient. It uses Canadian examples: the carriers, exchange points and regulators a designer works with in Canada.

A network of networks

The internet is not one network. It is more than 70,000 independent networks, each called an autonomous system (AS), that have agreed to exchange traffic. An autonomous system is a network run by one organization with one routing policy: an internet service provider, a cloud provider, a university, a content company or a large enterprise. Each has a unique AS number. The networks tell each other which blocks of IP addresses they can reach using BGP (Border Gateway Protocol), the same protocol Lesson 3 described inside fabrics. When you load a web page, your request may cross three or four autonomous systems, each passing it on according to its own business relationships.

Those relationships come in two basic kinds.

Transit

In a transit relationship, a customer network pays a provider to carry its traffic to and from the entire internet. The provider announces the customer's addresses to everyone it connects with and gives the customer routes to everywhere. Small and medium networks, enterprises and many data centre operators buy transit, usually from two or more providers for resilience.

At the top of the hierarchy are a few Tier 1 networks, such as Lumen, AT&T, NTT and Arelion, that can reach every destination on the internet without buying transit from anyone, because they peer with each other. Below them, Tier 2 networks both buy some transit and peer widely.

Peering

In a peering relationship, two networks exchange traffic directly, but only traffic between their own customers. Neither pays the other to reach the rest of the internet. Peering is often settlement-free: neither side pays, because both benefit roughly equally. It can also be paid, when one side benefits more. Peering reduces transit costs, shortens paths and lowers latency, because traffic goes directly from one network to the other instead of through a third party.

Content and cloud companies peer very widely. A large share of all internet traffic now flows directly from a few huge content and cloud networks into access networks, and a data centre that hosts both kinds of network becomes a natural place for them to meet.

Where networks meet

Networks interconnect physically in a few ways.

• Internet exchange points (IXPs): an IXP operates a shared switching fabric, often spread over several data centres in a city. Each member network connects one or more ports to it and can then peer with any other member over that single connection. An IXP lets a network peer with hundreds of others without running hundreds of separate cables.
• Private network interconnects (PNIs): when two networks exchange a lot of traffic, they connect directly with dedicated links, usually cross-connects within the same building. PNIs carry the largest flows between big content providers and access networks.
• Carrier hotels: certain buildings in major cities, often older downtown office towers converted decades ago, became the places where long-haul fibre routes converged. Because every network was there, every new network came too. These buildings are extremely dense with interconnection and are very hard to replicate.

In Canada, the largest exchange is TorIX in Toronto, one of the largest exchanges in North America by number of members. Others include QIX in Montreal, VanIX in Vancouver, YYCIX in Calgary, YEGIX in Edmonton, OttIX in Ottawa and MBIX in Winnipeg. Many are not-for-profit, and CIRA, the organization that runs the .ca domain, has supported the growth of Canadian exchanges so that Canadian traffic can stay in Canada instead of travelling through US cities. The country's best-known carrier hotel is 151 Front Street West in Toronto, which houses hundreds of networks; 555 West Hastings Street (Harbour Centre) plays a similar role in Vancouver.

Inside the building: the meet-me room

Inside a colocation data centre, carriers bring their fibre into one or more meet-me rooms (MMRs). Each carrier terminates its fibre on its own patch panels or in its own small racks in the MMR. Each customer's cage or suite also has fibre running back to the MMR. To connect a customer to a carrier, or two customers to each other, the data centre operator installs a cross-connect: a fibre patch cord, or a dedicated fibre run, between the two parties' panels.

Cross-connects are a recurring revenue stream for colocation operators, typically billed monthly per connection, and in a busy facility there may be tens of thousands of them. The operator keeps careful records of each one, because a single mislabelled fibre can take a customer offline. MMRs are high-security spaces with their own access control, separate from customer areas, and they need power, cooling and pathways like any other technical room. Large campuses usually have two MMRs on opposite sides of the building, each fed from a different entrance.

Carrier-neutral and carrier-specific facilities

A carrier-neutral data centre lets any carrier bring fibre into the building and lets customers buy from any of them. Competition between carriers keeps prices down and gives customers choice and resilience. Most large colocation providers are carrier-neutral, and the number of carriers "on net" in a building is one of the main things customers compare.

A carrier-owned data centre is run by a telecommunications company, and customers may be expected to use that carrier's services. Enterprise and hyperscale self-built data centres usually contract with a few carriers to bring diverse fibre to the campus and may also lay their own dark fibre to nearby colocation sites and to their other data centres.

What carriers sell

Data centre operators and their customers buy connectivity in several forms, from raw glass to finished internet service:

• Dark fibre: unlit fibre strands, leased (often for 10 to 20 years under an indefeasible right of use, or IRU) between two points. The customer installs its own optical equipment and can run as much capacity as that equipment allows. Hyperscale operators use dark fibre between their data centres in a metro area.
• Wavelengths: a dedicated optical channel on the carrier's DWDM system (Lesson 9), typically 10, 100 or 400 Gb/s, between two locations. The carrier operates the optics; the customer gets a transparent pipe.
• Ethernet private lines and Ethernet virtual circuits: layer 2 connections between sites, sold at a committed bandwidth.
• IP transit (dedicated internet access): a port onto the carrier's network with full internet connectivity, priced by bandwidth.
• MPLS VPNs and SD-WAN: private wide-area networks connecting an enterprise's offices and data centres.
• Cloud on-ramps: private connections into public cloud providers, such as AWS Direct Connect, Microsoft Azure ExpressRoute, Google Cloud Interconnect and Oracle FastConnect. Enterprises use them to reach the cloud with predictable performance instead of over the public internet. Colocation facilities that host on-ramps attract enterprise customers for that reason.

Burstable billing and the 95th percentile

IP transit is often billed on the 95th percentile of usage. The carrier measures the traffic on the port every five minutes for the whole month, sorts the samples, discards the highest 5% and bills at the highest remaining value. In a 30-day month there are \(30 \times 288 = 8640\) five-minute samples; the top 432 (36 hours' worth) are discarded.

The method lets customers burst briefly above their usual traffic without paying for it, while still charging for sustained use. Usually there is also a commit: a minimum amount billed every month whatever the usage. Traffic is measured in each direction and the higher of the two 95th percentiles is normally used.

Connecting the building

Getting fibre into a data centre is a construction project in its own right, and it is often on the critical path of the whole development.

Outside plant and entrances

Carriers reach a site through outside plant: fibre cables in underground conduits (ducts), on utility poles, or in trenches along roads and railways. At the property line, cables enter maintenance holes (manholes) or handholes, then run through the owner's conduits into a building entrance room. Good practice is at least two entrance points on different sides of the building, each fed by a separate conduit path from a separate street, so that a single excavator cannot cut every connection. The entrance conduits are usually a bank of several 100 mm (4 in) ducts, with spare ducts for future carriers.

In Canada, telecommunications is regulated federally by the CRTC (Canadian Radio-television and Telecommunications Commission) under the Telecommunications Act. Carriers that want to build on public roads need the consent of the municipality, and the Act lets the CRTC settle the terms when a carrier and a municipality cannot agree. On private land the data centre owner grants easements. Major carriers include Bell, Rogers, Telus, Cogeco and Vidéotron, along with regional and wholesale fibre providers; a designer typically asks several to quote for diverse routes to the site early in planning.

Bringing new fibre to a greenfield site can take a year or more, especially if it involves permits for road crossings, railways, rivers or highways. It is one of the reasons site selection (DCE-410) looks closely at existing fibre routes.

Route diversity and shared risk

Two connections are only independent if they share nothing that can fail. Experienced designers check for shared risk: places where two "diverse" routes actually run in the same conduit, cross the same bridge, pass through the same carrier building or follow the same railway corridor. Two carriers may even lease fibre in the same cable from a third party. Carriers can provide route maps (often under a non-disclosure agreement) so that the owner can check physical separation.

The same reasoning (series and parallel availability, from DCE-200) applies to entrance rooms, meet-me rooms, and the routers and power feeding them. A building with two carriers that both terminate in the same meet-me room on the same power feed has a single point of failure in the middle.

Latency and distance

Light in optical fibre travels at about two-thirds of its speed in a vacuum: roughly 204,000 km/s, which is about 4.9 µs per kilometre. Fibre seldom follows a straight line; it follows roads and railways, so the route is often 1.2 to 1.5 times the straight-line distance. Every millisecond matters to some applications: financial trading, online gaming, real-time collaboration and synchronous storage replication.

This physical limit shapes how cloud providers lay out their regions. Data centres in one availability zone, or a few zones in one region, are kept within roughly 100 km of each other so that synchronous replication stays within about a millisecond of round-trip time, while being far enough apart that a single local disaster is unlikely to hit all of them.

Interconnection as a design driver

For a colocation operator, connectivity shapes the building: how many meet-me rooms and where, how many entrance conduits, how much space is reserved for carrier racks, and how fibre pathways run from the MMRs to every customer suite. For a hyperscale campus, it shapes the site: diverse fibre routes to the campus, dark fibre to other campuses in the region, and enough duct capacity for decades of growth. For an AI training campus, outside connectivity may be modest compared with the enormous internal fabric, but it still needs diverse routes for data ingestion, management and inference traffic.

In every case, the engineer's job is to make sure that the physical paths, rooms, power and security support the network relationships the business depends on, and that no single conduit, room or power feed can sever them.