Seventeen Hops to 1.1.1.1

My laptop’s entire knowledge of the internet is one line.

$ netstat -rn -f inet
Destination        Gateway            Flags    Netif
default            192.168.12.1       UGScg    en0
192.168.12         link#15            UCS      en0

The second line says “addresses starting 192.168.12 are on my Wi-Fi, talk to them directly”. The first says “everything else: hand it to 192.168.12.1 and stop thinking”. That is the whole design of internet routing in miniature. No machine knows the way to everywhere. Each one knows a little and knows who to give up to. The previous post followed one message down the layers and waved at the part between the ISP router and the server. This is that part: what a packet meets between my default route and a machine on the other side of the world, and how every box along the way decided where to send it.

What the road actually looked like#

traceroute is the tool for seeing the road, and here is a real one from my desk to 1.1.1.1, Cloudflare’s public DNS resolver. It is uglier than the textbook version, which is exactly why it is useful.

$ traceroute -m 18 -w 1 -q 1 1.1.1.1
 1  kvd21 (192.168.12.1)         9.668 ms
 2  192.0.0.1                    8.875 ms
 3  192.0.0.1                   74.208 ms
 4  192.0.0.1                   65.937 ms
 5  192.0.0.1                   58.078 ms
 6  192.0.0.1                   30.483 ms
 7  10.177.63.130               30.128 ms
 8  10.177.63.112               40.304 ms
 9  10.180.163.67               31.588 ms
10  10.160.63.249               24.475 ms
11  10.160.107.100              90.371 ms
12  10.177.118.5                83.953 ms
13  *
14  172.68.188.42               73.240 ms
15  172.68.188.121              50.069 ms
16  172.68.188.139              54.260 ms
17  one.one.one.one (1.1.1.1)   32.197 ms

Seventeen routers, and only the last four belong to the destination. Reading it top to bottom:

• Hop 1 is my home router, the only address my laptop was ever told about.
• Hops 2 to 6 all answer as 192.0.0.1. My home internet is cellular, and those are five routers inside the carrier’s radio network that all borrow the same special-purpose address to reply from. The packet is still not on the public internet.
• Hops 7 to 12 are 10.x.x.x addresses, which are private, the same kind your office uses. Six more routers, all inside the carrier. Twelve of the seventeen hops happen before my packet has left one company.
• Hop 13 is a router that received the packet, forwarded it, and declined to say so. The * is not a failure of the path, only of the answer.
• Hops 14 to 16 are 172.68.x.x, Cloudflare’s network. The packet has crossed exactly one border: from my carrier straight into the destination’s network, with no one in between.
• The times do not add up. Hop 3 reports 74 ms and hop 17 reports 32 ms, yet the packet passed through hop 3 on its way to hop 17. Each line is a separate measurement of how quickly that router’s CPU bothered to write a reply, not a measurement of the path. Routers forward packets in dedicated hardware and answer questions about themselves in software, as an afterthought.

The trick traceroute plays#

Nothing in a router announces its presence. Traceroute discovers them by abusing the TTL, the hop counter from the IP header that every router decrements. When a router decrements a TTL to zero it throws the packet away and, as a courtesy, sends a small ICMP message back to the sender saying “time exceeded”, from its own address. Traceroute sends a packet with TTL 1, then TTL 2, then TTL 3, and collects the courtesy notes.

sequenceDiagram
  participant L as laptop
  participant R1 as router 1
  participant R2 as router 2
  participant D as 1.1.1.1
  L->>R1: probe, TTL 1
  Note over R1: TTL becomes 0: drop it
  R1-->>L: ICMP time exceeded, from 192.168.12.1
  L->>R1: probe, TTL 2
  R1->>R2: same probe, TTL 1
  Note over R2: TTL becomes 0: drop it
  R2-->>L: ICMP time exceeded, from 192.0.0.1
  L->>R1: probe, TTL 17
  R1->>R2: TTL 16
  R2->>D: TTL 1, delivered
  D-->>L: port unreachable: I am the destination

The router at hop 13 simply has ICMP replies turned off or rate-limited, so its note never comes and traceroute prints * after waiting. The RTT next to each hop is the time until the note arrived, which is why a slow-to-answer router looks “far” even when it is not. None of this touches the packet’s real path.

What every box actually knows#

Every router, from the one in my living room to the ones carrying a continent’s traffic, does the same thing with every packet: look at the destination address, find the most specific matching row in a table, send the packet out the port that row names. The table is a list of prefixes (an address block written as a network and a length, like 203.0.113.0/24 for the 256 addresses starting 203.0.113) and the rule is longest prefix match: when several rows match, the one with the longest length wins, because it is the most specific claim.

My laptop’s table has two rows. My home router’s has a handful. A router in the middle of the internet, one with no default route because there is nobody left to give up to, carries a row for every address block anyone on earth has announced:

A million rows, looked up for every packet, at tens of millions of packets per second per port. That is not done by software walking a list. Core routers hold the table in special memory (TCAM, a chip that compares a value against every stored row at once) and answer in a few nanoseconds. The chip has a fixed size, and on 12 August 2014 the IPv4 table crossed 512,000 rows, which was the default capacity of a very common line of routers. Routers all over the world began dropping routes or crashing, and operators spent the day reallocating memory. The table has doubled since.

Two words for the same box are worth separating here, because the traceroute above was about their difference. The control plane is the software that learns routes and builds the table; it is slow, runs on a general CPU, and is what answers ping and traceroute. The data plane is the hardware that looks up and forwards; it never stops for anything. A router can be forwarding a hundred gigabits per second flawlessly while its control plane is too busy to say hello.

The packet, station by station#

Here is the packet from my traceroute again, at each place it was rewritten or redirected, with the fields that changed since the previous station in orange. Two things I did not expect before I looked: the packet’s source address is rewritten twice before it reaches the public internet, and the destination network is entered directly from my carrier, with no intermediary at all.

default            192.168.12.1
192.168.12.0/24    on en0 (Wi-Fi)

NAT, twice. My laptop’s 192.168.12.23 is a private address, meaningless outside the house, so the home router swaps it for its own address and remembers the swap in a table keyed by port, so the reply can be un-swapped. My carrier then does the same thing again at a much larger scale: the home router’s address is itself from a shared range (100.64.0.0/10, set aside for exactly this), and a carrier-grade NAT maps thousands of homes onto a smaller pool of public addresses. This is what running out of IPv4 addresses looks like in practice. It also means the “IP addresses stay the same end to end” rule from the last post holds only across the public internet; at both edges it is bent.

Leaving one network. Inside the carrier, the twelve routers do not use BGP to talk to each other. They run an interior protocol, OSPF or IS-IS, which floods a map of the carrier’s own links so every router can compute shortest paths within the company. BGP enters at the border, where the question changes from “which of my routers is closest to the exit” to “which exit, and which other company”.

Networks are the unit, not routers#

From the outside, the internet is not routers. It is about 78,000 autonomous systems (an AS is one organisation’s network under one routing policy, identified by a number), and a packet crossing the internet is a packet crossing AS boundaries. My carrier is one AS. Cloudflare is AS 13335. A university is an AS; so is a bank with its own address space; so is your company if it has ever bought its own IP block.

What connects them is money. Two ASes with a link between them have one of two relationships:

• Customer and provider. The customer pays. The provider carries the customer’s traffic to and from the whole internet. Most of the 78,000 are pure customers: 66,500 of them have no customers of their own.
• Peers. Neither pays. Two networks with roughly balanced traffic connect directly, often at an internet exchange (an IXP: a building with a very large switch where hundreds of networks plug in and peer with each other for a flat fee). There are more than 1,200 of these. My hop 13 to 14 was almost certainly one: my carrier and Cloudflare, peering.

At the top sit a dozen or so networks that peer with each other and pay nobody, the tier 1s. Everyone else reaches the rest of the world through a chain of providers that ends at one of them, unless a shortcut exists through a peer.

This relationship structure decides routes more than distance does. A network will always prefer to send traffic through a customer (who pays it) over a peer (free) over a provider (whom it pays), regardless of which path is shorter. And a network will never carry traffic from one provider to another provider, because it would be paying twice to move packets that make it nothing. The rule is called valley-free routing and it is the reason two nearby networks can have traffic between them travel through another city: neither is willing to be the free middleman.

Here are real routes to 1.1.1.0/24 as seen by six of RIPE’s route collectors on the day I wrote this, each written as the list of ASes the announcement passed through, nearest first, origin last:

7018 13335                     AT&T, then Cloudflare
15692 13335
61138 13335
18106 13335
835 13335
46997 38008 38008 13335        one network listed itself twice on purpose

Almost everyone is one hop from Cloudflare, which is what heavy peering looks like. The last line shows a network padding its own name twice into the path it announces to one neighbour, a trick called prepending: it makes that route look longer so other networks prefer a different door into it. The path is not a measurement. It is a message, and networks edit it to steer traffic.

Gossip with a receipt#

BGP, the Border Gateway Protocol, is how those 78,000 networks tell each other what they can reach. It is 1989’s design (RFC 1105, revised into the version still running today in 1994), and the mechanics are almost disappointingly small.

Two neighbouring routers in different ASes open a plain TCP connection to each other on port 179 and keep it open for years. Over it they send two kinds of messages: “I can reach this prefix, and here is the list of ASes a packet would pass through if you send it to me”, and “I can no longer reach this prefix”. When a router learns a route it may pass it on to its other neighbours, after adding its own AS number to the front of the list. A route that arrives with your own number already in the list has gone in a circle, and is dropped. That is the entire loop-prevention scheme.

Each router ends up with several candidate routes for each prefix, one per neighbour that offered one, and picks a single best. The order of the decision is long in the standard, but the part that matters is short:

Step 5 has a consequence that took me a while to accept: the route a packet takes to a server and the route the reply takes back are usually different. Each network gets rid of the packet at its nearest exit, and “nearest” is measured from wherever the packet happens to be. My traceroute to 1.1.1.1 shows one path. The replies came back another way, and there is no command that shows both. “The path” between two machines does not exist; there is a path there and a path back.

Watch the whole thing happen on a small internet of six networks. The dots on the links mark who pays whom. Step through the rounds and read each network’s table as announcements arrive; then switch on the hijack.

The hijack in the simulator is not hypothetical, and the more-specific trick is exactly how the most famous one worked. On 24 February 2008 Pakistan Telecom, meaning to block YouTube inside Pakistan, announced 208.65.153.0/24, a more specific slice of YouTube’s own 208.65.152.0/22. Its provider passed the announcement to the world without checking, longest-prefix match did the rest, and for about two hours the planet’s YouTube traffic flowed into Pakistan and died. YouTube’s fix was to announce the same /24 itself, then even more specific /25s, to win the match back.

One address, three hundred cities#

The last thing my traceroute revealed was in how short it was. 1.1.1.1 answered from 32 ms away. That is not because Cloudflare has a machine near me. It is because Cloudflare announces 1.1.1.0/24 from more than 300 cities at the same time, and BGP, picking the shortest AS path and then the nearest exit, delivers each user to whichever announcement is closest to them. The technique is called anycast, and it is why a single IP address can be “in” every country at once.

Anycast has a cost that follows directly from how it works: a TCP connection assumes both ends stay put, but if a route flaps mid-connection your packets can land at a different city that has never heard of your connection. For DNS, where each exchange is one packet, nobody notices. For long TCP connections it takes care, which is one reason CDNs work hard to keep routes stable.

When the table lies#

BGP has no notion of truth. A network announces a prefix and its neighbours believe it, because in 1989 the internet was a few hundred networks run by people who knew each other. The four incidents below are the ones I use to remember what can go wrong, because each is a different failure of the same trust.

The fix that exists is RPKI (Resource Public Key Infrastructure): the registries that hand out address blocks let the owner sign a statement saying “only AS 13335 may originate 1.1.1.0/24”, and routers can check announcements against those statements and drop the ones that fail. That is the “origin validation” toggle in the simulator. As of June 2026, 67% of announced prefixes are covered by such a statement, a record; but only 12% of networks actually enforce validation on everything they receive, and 36% do not validate at all. RPKI would have stopped the 2008 and 2019 incidents. It would not have stopped 2021, which was a valid network withdrawing its own valid routes, and it says nothing about the path: a network can still claim a short route to a correctly signed origin it has no connection to. Work on signing paths too (ASPA) is in progress and deployed almost nowhere yet.

The physics under the tables#

Every route in every table is ultimately a piece of glass. About 600 submarine cables, 1.5 million kilometres of them, connect the continents, and light in fibre covers about 200 kilometres per millisecond. That number sets a floor no routing decision can beat: New York to London is 5,600 km of cable, so 28 ms one way and 56 ms round trip at the very best, and real measurements land at 70 to 80. My 32 ms to 1.1.1.1 says the answering site was within roughly 3,000 km of cable, and probably far less, with most of the time spent in the carrier’s own network rather than in the glass.

Which is the last surprise of the traceroute. Twelve of seventeen hops, and most of the latency, were inside one company before the packet had touched the internet at all. The public internet was one border crossing and four routers. The picture I carried for years, of a packet bouncing across a dozen independent networks to reach anything, was a picture of 1995. Today’s packet typically leaves its access network and lands in the destination’s network directly, because the large destinations peer with everyone at the exchanges, and they announce their addresses from everywhere.

The tables that make this work still rest on the same act of faith they did in 1989: a network says what it can reach and its neighbours believe it. RPKI has moved two thirds of the prefixes onto signed ground and one eighth of the networks onto checking it. What I do not know is whether the remaining distance closes in a decade, or whether the internet keeps working the way it always has, on the fact that most of 78,000 organisations are, most of the time, telling the truth.