Latency Numbers and Throughput Ceilings: The Constants Behind Capacity Estimation

Part 1 built the four-step method, and its step 2 — divide the load by what one machine can do — has a dirty secret: you cannot divide by a number you don’t have. When someone freezes at “how many database servers?”, it’s almost never the division that’s missing. It’s the denominator.

This post is the denominators. Not a phone book of benchmarks — a small set of latencies and ceilings, chosen because everything else can be derived from them. By the end you should be able to answer “roughly how many X can one node do?” for an X you’ve never benchmarked, and show your work.

The ladder: four bands, ~100× apart#

Computers wait on four kinds of places, and the tragedy of systems engineering is how far apart they are. Think of it as geography:

The classic version of this is Jeff Dean’s “latency numbers every programmer should know.” But reading it as a static table undersells it — the point of these numbers is their ratios, and ratios are best felt. Click any operation below to make it the anchor and watch how many of everything else fits inside one of it. Then flip on human scale, where a 1 ns L1 hit becomes one second — and a cross-continent packet becomes 4.8 years:

One row always earns a double-take, so let’s decode it: “disk” on this ladder means a spinning HDD — a mechanical arm over rotating platters — while the SSD rows are flash. Read the rows again with that in mind and a legendary inversion appears: the datacenter network (~500 µs) is faster than a local spinning disk (825 µs for 1 MB; a seek — the arm physically moving — costs 2–10 ms, capping an HDD at ~100–200 random reads/sec). That inversion is why fetching another machine’s RAM (Redis, memcached) beats reading your own mechanical disk, and why Kafka runs happily on cheap HDDs by refusing to ever seek — append and stream only. The modern footnote: NVMe SSDs (49 µs per MB) beat the network again, so today’s order is RAM > NVMe > datacenter hop > spinning disk.

The wire in Grace Hopper’s hand#

Long before anyone made gists of these numbers, Grace Hopper made them physical. At lectures she handed out lengths of wire cut to 11.8 inches — the distance light travels in one nanosecond. When admirals asked her why satellite links were slow, she’d hold one up: this is a nanosecond; physics sets the floor, not lazy engineers. For contrast she kept a 984-foot coil — one microsecond — and liked to say it should hang over every programmer’s desk, “so he knows what he’s throwing away when he throws away a microsecond.” (Her nanoseconds are in the Smithsonian now.)

Extend her trick one more rung and it lands on this series’ favorite constant: a millisecond of light is 186 miles of wire — the fsync that guards every database commit, stretched from one city to the next.

The five numbers to memorize (and the formula that replaces the rest)#

You do not need the whole ladder cold. Five rungs cover practically every capacity argument:

One caution before carving these into stone: the rungs drift, and at different speeds. Colin Scott’s interactive version of this table replays every number from 1990 to today — NIC bandwidth doubles every couple of years, DRAM keeps creeping, and a disk seek has barely moved in two decades. So the absolutes age, but the ratios are durable knowledge — which is exactly why the ladder interactive above is built around ratios, not values.

For everything else, don’t memorize — derive. The universal derivation tool is embarrassingly simple:

A core is a budget of one million microseconds per second. Estimate how many microseconds one operation costs, divide, done:

• A Redis GET is a hash lookup in RAM plus network handling — call it ~10 µs of work → $10^6 \div 10 =$ ~100K ops/sec on one core. That’s the famous Redis number, derived.
• Parsing a small JSON message ≈ ~5 µs → ~200K parses/sec/core — which is why a market-data consumer saturates a core long before its NIC (Part 1’s two-ceilings story, now with a formula behind it).
• A request that does ~1 ms of real work (some parsing, a RAM-heavy computation) → ~1K/sec/core, so a 16-core node gives ~16K/sec. Suddenly “a few thousand QPS per app server” stops being folklore and becomes arithmetic.

This formula is the difference between knowing constants and owning them. When a number you half-remember comes up, you re-derive it in five seconds and trust yourself.

Why every durable write costs a millisecond#

Number 3 on the list — fsync ≈ 1 ms — deserves its own section, because it single-handedly explains database performance.

A database’s core promise is durability: once it says “committed,” the data survives a power cut. The only way to keep that promise is to write to disk and wait for the disk to confirm — that confirmed write is the fsync (the database’s write-ahead log does exactly this before anything else happens). Think of it as a notary: any agreement is legally void until the notary stamps it, the stamp takes a fixed ~1 millisecond, and there is exactly one notary per database.

Do the napkin math on that: 1 ms per stamp → ~1,000 stamps per second. If every row insert demands its own stamp, your thousand-dollar database server inserts a whopping one thousand rows per second — a number that shocks people the first time they see it, and it’s not a bug. It’s physics plus a promise.

The escape is not a faster notary — it’s carpooling: put 100 rows in one appointment, get one stamp for all of them. Same disk, same millisecond, 100× the throughput. Drag the lever and watch — including what happens when carpooling works too well:

Three things this widget quietly teaches:

• The naive ceiling (~1K/s) and the batched ceiling (~100K+/s) are the same machine. When someone quotes “Postgres does 5–10K inserts/sec,” that’s partial carpooling already (databases group concurrent commits — group commit — without being asked). Explicit batching or COPY buys the rest.
• Every fix relocates the bottleneck. Past ~300 rows per batch the disk idles and the CPU (parsing, validating, updating indexes) becomes the wall. Capacity work is not removing bottlenecks — it’s choosing where the bottleneck lives.
• Latency pays for throughput. A row in a 100-row batch waits for 99 companions before its stamp. For telemetry, nobody cares; for a payment confirmation, you do not batch across the user’s promise. What’s batchable is a product question wearing an engineering costume.

This exact napkin argument has a famous field test. Simon Eskildsen’s napkin-math project prices MySQL’s insert ceiling from first principles — ~1,000 fsyncs/sec, therefore ~1,000 transactions/sec — then measures 5,300/sec on real hardware. The 5× gap isn’t the napkin failing; it’s the napkin discovering group commit. His rule is worth adopting whole: napkin and reality should agree within an order of magnitude — a bigger gap means a bug or an optimization you haven’t heard of yet. Estimation isn’t a ritual; it’s a production debugging tool. (And an honesty note on the constant itself: a tuned local NVMe can fsync in ~300 µs while a cloud-network disk takes 1–2 ms — the exponent is the anchor, not the digit.)

The vocabulary: per-node ceilings worth carrying#

With the ladder and the derivation formula in hand, the standard ceilings stop being trivia and start being consequences. Every row below is explained by the rungs above it:

Two reading rules for this table. First, quote exponents, not decimals: say “order of 10K, call it 5–10K” — false precision reads worse than honest rounding. Second, remember Part 1’s headroom rule: you provision at 60–70% of any ceiling, because queues near saturation don’t degrade politely — push a system from 70% to 95% and p99 latency doesn’t rise 35%, it roughly quadruples (measured on my own feed pipeline: +12% throughput took p99 from ~8 ms to ~30 ms). The last third of a ceiling is a trap, not spare capacity.

Interlude: two ceilings per box, always#

A ceiling question that sounds like one number is usually two, and the smaller one wins. A WebSocket node has a connection ceiling (~25K, set by memory and file descriptors) and a send-rate ceiling (~40K/s, set by CPU per message). A quiet chat app with a million idle connections hits the first; a sports-score fan-out with few viewers but frantic updates hits the second. Same server, opposite failures — you must check both, exactly like Part 1’s event-rate vs byte-rate split. This “compute both, take the worse” reflex is the whole trick of the drills at the series’ end.

And the bookkeeping behind “how much sits in flight at once” has a name worth knowing: Little’s Law — concurrency = throughput × latency. A node serving 10K requests/sec at 100 ms each is holding 10K × 0.1 = 1,000 requests in flight at every instant, and each one occupies a socket, a thread or task, and some buffer memory. That one multiplication sizes connection pools, worker counts, and queue depths — the resources that are otherwise invisible until they run out. (It only holds for steady averages — spikes and synchronized bursts break mean-based reasoning, per Part 1. And the derivation formula above is the same identity rearranged, with concurrency pinned at one core’s worth.)

The pocket card#

• Memorize five numbers: RAM 100 ns, SSD 100 µs, fsync 1 ms, datacenter RTT 0.5 ms, cross-region 50–150 ms. Rungs, not trivia.
• Derive the rest: one core = $10^6$ µs/sec; divide by the microseconds one operation costs.
• fsync is the notary: ~1K stamps/sec naive; batching is carpooling; latency pays for throughput.
• Every fix relocates the bottleneck — after each lever, re-ask “what’s the wall now?”
• Quote exponents (“order of 10K”), and provision at 60–70% — the knee is real and it is not linear.
• Check both ceilings per box (connections and rate; rows and bytes) — the smaller one is the truth.

All of these ceilings meet their hardest test in one place: the database — the only tier where a copy can’t quietly take work off the original.