Internet routing, content delivery and the enterprise WAN

Lesson 7 described the business of connecting networks: transit, peering, exchange points and meet-me rooms. This lesson looks at how traffic actually finds its way once the connections exist. It covers how BGP chooses routes across the internet and how operators steer traffic with it, why routing is vulnerable to mistakes and attacks and how it is being secured, how DNS and content delivery networks bring content close to users, and how enterprises connect their offices and data centres over wide-area networks.

The internet as autonomous systems

The internet is a network of more than 75,000 independently operated networks called autonomous systems (ASes), each identified by an AS number (ASN) and each controlling a set of IP address blocks (prefixes). Carriers, cloud providers, content companies, universities and large enterprises all run their own ASes. Regional internet registries assign addresses and ASNs: ARIN serves Canada, the United States and parts of the Caribbean.

ASes exchange reachability information using the Border Gateway Protocol (BGP). Each AS announces the prefixes it originates to its neighbours, and each neighbour passes the announcement on, adding its own ASN to the front of the AS path. A router that receives the same prefix from several neighbours chooses one best path and installs it in its forwarding table. The global table held by internet edge routers contains roughly a million IPv4 prefixes and over 200,000 IPv6 prefixes (Lesson 6).

How BGP chooses a path

BGP is a policy protocol: it does not simply pick the fastest path, it picks the path the operator's policy prefers. When several paths to the same prefix are available, a router compares them in a fixed order and stops at the first difference. Simplified, the common order is:

1. Highest local preference: a value the operator sets inside its own AS to express which exit it prefers, such as customers over peers over transit providers, because customers pay and transit costs money.

2. Shortest AS path: fewer ASes to cross.

3. Lowest origin type, then lowest multi-exit discriminator (MED), a hint from a neighbour about which of several links into it to use.

4. Paths learned from external neighbours (eBGP) over internal ones (iBGP).

5. Lowest internal routing cost to the exit point (hot-potato routing: hand the traffic off as soon as possible).

6. Tie-breakers such as the oldest path or lowest router ID.

BGP does not see latency, congestion or packet loss. A path that is short in AS hops may be long in kilometres, which is one reason large content providers add their own measurement-driven traffic engineering on top of BGP.

Steering inbound traffic

Local preference controls how traffic leaves an AS. Controlling how traffic arrives is harder, because other networks make that choice. Operators use several tools:

• AS-path prepending: announcing a prefix with its own ASN repeated, such as "64500 64500 64500", over a less preferred link so that the path looks longer to others.
• More-specific prefixes: announcing a /24 over one link and the covering /23 over both. Routers always prefer the most specific matching prefix, so traffic for the /24 arrives on the first link while the /23 provides a backup.
• BGP communities: tags attached to announcements that ask a provider to take actions, such as lowering local preference in its network, not announcing the route to certain peers, or dropping traffic to an attacked address (blackholing, Lesson 4).

When routing goes wrong

BGP was designed in an era of trust: by default a router believes what its neighbours announce. That makes the internet vulnerable to two kinds of incident:

• Route hijacks: an AS announces prefixes it does not own, accidentally or deliberately, attracting traffic meant for someone else. In February 2008 Pakistan Telecom, trying to block YouTube domestically, announced a more-specific route for YouTube's addresses; it leaked to the world and took YouTube offline globally for about two hours.
• Route leaks: an AS passes routes learned from one provider or peer to another, against policy, so that traffic is pulled through a network that cannot carry it. In June 2019 a small ISP in Pennsylvania leaked thousands of more-specific routes, generated by route-optimization software, to a large transit provider that accepted and propagated them, disrupting access to major services for hours.

Defences have improved steadily:

• Prefix filtering: providers accept from each customer only the prefixes that customer is registered to announce, built from internet routing registry data.
• RPKI (Resource Public Key Infrastructure): address holders publish signed Route Origin Authorizations (ROAs) stating which ASN may originate each prefix. Routers performing route origin validation reject announcements that conflict with a ROA. By 2025 about half of IPv4 routes were covered by ROAs, and most large networks reject invalid routes.
• Maximum-prefix limits shut down a session if a neighbour suddenly announces far more routes than expected.
• MANRS (Mutually Agreed Norms for Routing Security) is an industry program in which networks commit to filtering, anti-spoofing, coordination and validation.

RPKI validates who may originate a prefix but not the rest of the path; work on path validation (such as ASPA, Autonomous System Provider Authorization) continues.

DNS and anycast

Users reach services by name, so DNS (Lesson 4) is the first step of nearly every connection. A user's device asks a recursive resolver (run by the ISP, the enterprise or a public service), which queries the root servers, then the servers for the top-level domain (such as .ca or .com), then the domain's authoritative servers, caching each answer for its time to live (TTL).

DNS is a natural fit for anycast: the same IP address is announced by BGP from many locations, and each user's traffic goes to the nearest one in routing terms. The 13 root server identities (A to M) are served by more than 1,500 anycast instances around the world, and large public resolvers and authoritative DNS providers work the same way. Anycast spreads load, shortens distances and absorbs DDoS attacks, because each instance takes only the attack traffic from its own region.

DNS is also a traffic-steering tool. Authoritative servers can return different addresses to different users, directing each to a nearby or healthy data centre (global server load balancing). Short TTLs let operators shift traffic quickly during a failure, at the cost of more queries.

Content delivery networks

A content delivery network (CDN) is a distributed system of cache servers placed in many cities, often inside ISPs' networks and at exchange points, that stores copies of popular content close to users. When a user requests an image, video segment or software update, the CDN serves it from a nearby cache if it has it (a hit) and fetches it from the origin data centre if it does not (a miss).

• Users get lower latency and higher throughput, because the content travels a short distance (Lesson 2).
• The origin data centre serves only the misses, so it needs far less capacity and transit.
• ISPs save on long-haul and transit capacity by hosting caches inside their networks.

Most internet traffic by volume, especially video, is now delivered from CDNs and from the caches that the largest content providers place directly in ISP networks. The same edge sites increasingly run application code, security services and DDoS protection (Lesson 4).

The enterprise WAN

Enterprises connect offices, factories, stores and data centres with wide-area networks (WANs). Two approaches dominate.

MPLS VPNs

A carrier's MPLS (Multiprotocol Label Switching) network carries each customer's traffic in a separate layer 3 VPN: the customer's sites exchange routes with the carrier, which keeps them in a per-customer VRF (Lesson 4) and forwards packets by short labels across its backbone. MPLS VPNs offer predictable performance, QoS classes for voice and video, and service-level agreements, but are relatively expensive per megabit and slow to order and change.

SD-WAN

Software-defined WAN (SD-WAN) builds encrypted tunnels between sites over any mix of transports, typically ordinary broadband internet, fixed wireless or cellular, and sometimes MPLS. Edge devices at each site measure the latency, jitter and loss of every path continuously and steer each application over the best one, failing over in under a second. Central controllers push policy to all sites. SD-WAN has largely displaced pure MPLS for branch offices because it is cheaper, quicker to deploy and can send cloud-bound traffic directly to the internet instead of back through a central data centre.

Enterprises increasingly combine SD-WAN with cloud-delivered security services (an approach often called SASE, secure access service edge), so that branch and remote users connect to nearby security points of presence rather than back-hauling traffic to a central firewall.