Building a Real-Time Ad Bidder in Rust — Part 4: Under Load

Everything this series has described so far is a claim. Parts 1–3 designed a hot path, guarded the money, and served a model — all of it asserting, implicitly, that the system behaves under pressure. This part is the trial. It has two acts: first the machinery built for bad days — circuit breakers, hedged reads, bounded event queues — and then the load tests that judged all of it, which found the real villain somewhere none of that machinery was pointing.

Act one: designing for the bad day#

A slow Redis isn’t a broken Redis — treat it as one anyway#

The classic circuit breaker counts errors: too many failures, stop calling the dependency, probe later, recover. For a latency-bound system that’s only half the disease. A Redis that answers every call in 30ms is returning zero errors and destroying you — every request burns most of its deadline waiting, then misses the SLA. So this breaker (bidder-core/src/breaker/) is latency-aware: it opens when the error rate or the slow-call rate crosses 50% over a minimum of 20 calls. Slowness is failure. When open, dependent stages skip their lookup and degrade the way Part 1’s failure table prescribed — bid without the cap check, count the skip — and a half-open state lets exactly one probe through to test recovery.

Two bugs caught in review are worth more than the happy path:

• Per-call-site breakers were independent. The frequency capper and the segment repository each wrapped the same Redis with their own breaker — so Redis slowness observed by one didn’t protect the other. One dependency, one breaker, shared by every call site. A breaker per call site isn’t isolation; it’s amnesia.
• The half-open probe had a race. Two concurrent requests could both observe “no probe in flight” and both get through. Classic check-then-act; the fix is doing the check-and-set under one lock. Rust’s ownership rules prevent data races, not logic races — no language saves you from TOCTOU.

Hedged reads, and the bug where the cure became the disease#

Tail latency on a remote read has a well-known mitigation, canonized in Dean and Barroso’s The Tail at Scale: if the read hasn’t returned by roughly its p99, fire a duplicate and take whichever answers first. The slow original usually lost to a fast retry against a healthier replica or a luckier moment.

The catch: a hedge is deliberate extra load on a dependency that may be slow because it is overloaded — a feedback loop wearing a cape. So every hedge here has to pass four gates before it fires: a token budget (hedges per second are capped), the breaker must be closed, the trigger only fires past a floor (8ms), and a kill switch disables hedging entirely whenever the system is shedding load. Resilience features need their own backpressure, or they become the incident.

And one did — almost. In review, the original implementation turned out to issue the duplicate call on every hedged read, not just slow ones: the timeout wrapper dropped the original future when the timer fired, so the code then always issued a second call — two calls minimum, three on the slow path. A mechanism meant to shave the p99 tail was quietly multiplying Redis load. The fix was Rust-specific and instructive: pin the original future and race it against the timer with select!, so it stays in flight across the trigger boundary and the duplicate fires only when genuinely needed. In async Rust, who owns the future is the correctness question; a dropped future is a cancelled call, whether you meant it or not.

Events: bounded, prioritized for honesty under fire#

Every bid and win emits an analytics event to Kafka — strictly off the hot path. Publishing is a synchronous, non-blocking enqueue into the producer’s bounded queue (100,000 entries), with a dedicated thread handling delivery callbacks: the hot path never waits on a broker, so Kafka slowness can never add a millisecond to bid latency. The interesting design is the drop policy when the queue fills. Dropping the newest is cheapest — but during a sustained incident it produces a dataset that ends exactly when things got interesting. So there’s an incident mode: a monitoring loop watches the drop rate, and when drops stay elevated it auto-flips the policy to random sampling (and auto-reverts after recovery). A uniformly thinned stream keeps every time period represented, so post-incident analytics see a fair sample of the disaster instead of a clean recording that stops at its start. Losing data is survivable; losing it with a bias is how you draw wrong conclusions afterward.

One more review catch that belongs to the money story: events were being spawned before their serialization was checked, so a serialization failure returned an error to the caller while phantom events were already in flight — and impressions were being recorded on both the bid and the win handler, double-counting every win. Neither bug is exotic. Both are invisible until someone reconciles two numbers that should match. The cheapest resilience machinery in this whole section was the code review.

Act two: the verdict#

Making the test honest before trusting it#

Three lessons about load-testing itself came before any bidder numbers, all transferable:

• Compressed time breaks realistic money. A 3-minute run at 5K RPS simulates hours of spend. With production-realistic daily budgets ($50–$5,000), every campaign exhausted in the first ~30 seconds — and the “load test” measured a system that mostly answers “budget exhausted.” Budgets in the seed corpus had to be inflated to keep the bid path exercised. If your test compresses time, every time-denominated quantity in the system needs rescaling, or you’re measuring the wrong regime.
• The 17-minute request. The first run reported one request taking 17 minutes. Not the bidder: a missing per-request timeout in the load script meant a stuck TCP socket sat until macOS’s retransmit timer gave up — ~15–17 minutes — wrecking the summary statistics. Distrust the harness before the system.
• The harness has capacity math too. When a breaker trip briefly spiked latency, the load generator’s small pre-allocated worker pool starved (5,000 RPS × 80ms stall = 400 workers needed; 250 allocated) and the tool under-delivered load while appearing to show bidder slowness. The generator needs headroom for the system’s worst moment, not its average.

The baseline that pointed a finger#

With the harness honest, the baseline (5K and 10K RPS) landed inside SLA — p99 of 2–9ms — but the per-stage arithmetic told a sharper story. No profiler: Prometheus publishes each stage’s duration histogram as a running sum and count, and sum ÷ count is the average cost per call. Two curls and a division:

frequency_cap:        330.13 s / 319,262 calls → 1,034 µs per call
candidate_retrieval:   98.15 s / 319,262 calls →   307 µs per call
budget_pacing:          3.61 s / 319,262 calls →    11 µs per call

One stage — frequency capping — was 75.7% of all pipeline time. Part 1’s cost model predicted the suspect class (it’s the per-candidate I/O stage); the histograms convicted it. And a second counter said why: ~7.86 million cold cache misses across 319K requests — 24.6 Redis round-trips per request from a stage that was supposed to be mostly cache hits.

The 112x fix#

The cache in front of the counters held a per-campaign map under each user — but on a miss for one (user, campaign) pair, the code fell through to Redis, used the answer, and never put it in the cache. Every future request for that pair missed again. The fix is three careful moves, not one:

1. Insert the Redis result into the cache on fallback — the obvious part.
2. With the real counts, not zeros — inserting an empty entry would tell every future request “this user has seen nothing,” quietly breaking the cap.
3. With max() merge semantics — a concurrent flow may have already written a higher count; a blind overwrite would un-count impressions.

Plus a new counter to prove the warm path was actually firing — fixes to invisible problems need their own visibility.

The numbers, and their caveats#

With the cache in place, the stress tiers:

20,000 to 75,000 requests per second at p99 between 3.4 and 4.3ms, zero errors across 6.1 million stress-tier requests, bid rate flat at ~70% throughout. An order of magnitude inside the 50ms SLA. Here’s the 75K run’s actual output, gates and all:

The 50K run’s output, for comparison — every gate green

Now the caveats, stated with the same confidence as the numbers, because they’re what make the numbers meaningful:

• One machine ran everything — bidder, Postgres, Redis, and the load generator, sharing cores and memory bandwidth on a single laptop. Production separates these; it swaps CPU contention for real network latency (~0.5–1ms per Redis hop), a different cost shape, not a free upgrade.
• 75K is the rig’s ceiling, not the bidder’s. The 100K attempt died in the load generator (ephemeral ports); requests that got through still answered at p99 ≈ 5ms with zero bidder-side errors. Past 75K on this hardware, the measurement is of the OS and the harness.
• The win is the caching architecture, not the language. The same in-process TinyLFU pattern exists in Java (Caffeine — same author). Rust’s honest contribution is narrower: no GC pauses inside the tail budget, lower allocator pressure, and a compiler that made the concurrent parts boring. The 112x came from a cache key, not from Rust.

The trade the cache demands#

The in-process cache that made these numbers possible has a distributed-correctness price, and it’s disclosed in the config rather than discovered in production: each pod’s cache is its own view. Two pods behind a round-robin balancer can independently approve the same impression — a cap of N becomes up to N × pods during bursts. So the cache ships disabled by default: turning it on is an explicit choice among sticky per-user routing, an SLA that tolerates approximate caps, or a single instance. A default that silently violated advertiser contracts in the most common deployment shape would be a trap, not a feature. Deployment shape is part of correctness, and a config flag is sometimes the honest way to say so.

What to take away#

1. Breakers must treat slowness as failure, and one dependency gets one breaker — shared by every call site that touches it.
2. Any mechanism that adds load to help latency needs its own governor — token budgets, health gates, and a kill switch tied to load shedding. And in async Rust, dropping a future cancels its call: ownership is the correctness model.
3. When you must lose data, choose the bias. Random sampling under pressure keeps incident analytics fair; drop-newest ends the recording exactly when it matters.
4. Instrument stages, then do division. A per-stage histogram’s sum ÷ count found a 75%-of-pipeline bottleneck with two curls — before any profiler was installed.
5. Report caveats with the same rigor as results. One machine, a synthetic corpus, a harness with its own bugs — the numbers are real because the limits are stated.

That closes the series where it started: the intro promised a machine that answers in 50 milliseconds, and the measured answer came back at four. The parts left deliberately unbuilt — the segment ingestion platform, a trained and calibrated model, bid shading fed by win-notice outcomes — are each their own journey, and the trait boundaries in this codebase are where they’d bolt on. If you build one, the disclaimer below means what it says: corrections and war stories are welcome.

References & further reading#

• The Tail at Scale — Dean & Barroso; the canonical case for hedged requests and tail-tolerance
• Handling Overload — Google SRE book; load shedding and degradation as first-class design
• CircuitBreaker — Fowler’s original write-up of the pattern
• Programmatic advertising data flow — the per-stage latency budget these results are measured against

Disclaimer: I don’t work in ad tech. This series documents a personal project built to learn high-performance systems engineering, using real-time bidding as the problem domain because its constraints (hard latency SLAs, large working sets, high request rates) are unusually honest teachers. Industry references are from public engineering material; my own numbers come from synthetic workloads on developer hardware and are reported with their caveats. Corrections from people who build these systems for a living are very welcome.