Building a Real-Time Ad Bidder in Rust — Part 2: Where the Money Moves

Part 1 built the hot path; this part follows the money through it. Three of the nine pipeline stages exist purely to protect the advertiser’s wallet — budget pacing (don’t spend too fast), frequency capping (don’t spend it all on the same person), and the ranking/pricing/win-notice chain (pay the right amount, and count only wins that really happened). None of them is glamorous. All of them are the difference between a bidder and a money-shredding machine, because every mistake here is invisible in a latency dashboard and very visible on an invoice.

One ground rule first, so it never has to be said again: money is integer cents in this system, end to end. An i64 of cents survives every addition and comparison exactly; a float does not. If you build anything that touches money, this is the cheapest correctness decision you will ever make.

Budget pacing: the problem is time, not arithmetic#

A campaign says “spend at most $5,000 today.” Enforcing the amount is trivial — a counter. The interesting problem is when the money goes. Bid on every eligible auction from midnight onward and the budget is gone by mid-morning, because auctions arrive by the thousand per second. The advertiser paid for a day of presence and got a morning; everyone browsing in the evening — often the more valuable traffic — never sees the ad.

What this system implements: exact depletion, with an honest name#

The pacer here (bidder-core/src/pacing/) is a reserve-and-release depletion counter, and it’s worth describing precisely because its two subtleties transfer to any spend system:

• Reservation is atomic and pessimistic. At the pacing stage, each surviving candidate atomically decrements its campaign’s remaining budget (AtomicI64.fetch_sub — a read lock to find the counter, no write contention on the hot path). If the budget was already at zero, the candidate is dropped. A campaign the pacer has never heard of is treated as exhausted, not unlimited — misconfiguration should fail toward spending nothing, never toward spending everything.
• Losers give the money back. Reservation happens before ranking, but only one candidate per slot wins. After ranking, every non-winner’s reservation is released (bidder-core/src/pipeline/stages/ranking.rs). Skip this and budgets drain on bids that never happened — a leak that no single test catches, because each request looks correct in isolation.

One deliberate quirk: the check is “was there budget before this decrement,” not “is there enough for this bid” — so the last bid of a campaign’s day can overshoot by one bid’s worth. That’s a documented trade: the alternative (compare-and-swap loops on every candidate) costs more on the hot path than one bounded overshoot costs in money.

What the industry does on top: smooth pacing#

Depletion enforcement answers “did we overspend?” — it does not answer “did we spend well?” Production systems add a pacing policy on top, and the published literature is unusually good here. LinkedIn’s KDD paper describes the canonical approach: forecast the day’s traffic curve, give each campaign a spend schedule proportional to it, and probabilistically throttle participation (skip some auctions the campaign could have won) whenever actual spend runs ahead of schedule. Yahoo’s smart pacing work goes further: pace with feedback control, and preferentially drop the auctions with the lowest predicted response, so smoothing costs as little performance as possible.

The two families to know: throttling (bid less often, full price) and bid shading downward (bid always, lower price). Throttling is simpler and doesn’t distort the auction; shading interacts with pricing strategy. Most large systems use throttling for pacing and keep shading for the auction itself — which is the next section.

This project implements the enforcement layer and deliberately not the smoothing layer — the honest boundary: smoothing needs a traffic forecast, and a synthetic-traffic lab has nothing real to forecast.

Frequency caps: the semantics are a schema#

The intro post covered why caps exist (user experience plus a contractual promise). The engineering question is what a cap actually is, and the answer is: a schema decision. “At most N impressions per user” is underspecified until you fix two axes:

• What entity is capped — this campaign? this specific creative? ads on this device type? this daypart? Each is a real advertiser ask, and each is its own counter.
• Over what window — hour, day, week. Windows are independent counters, not one counter with math: an hourly cap of 2 and a daily cap of 5 are separate keys expiring on separate cadences.

So one user’s cap state is a small family of counters:

v1:fc:{u:42}:c:9001:d     campaign 9001, daily
v1:fc:{u:42}:c:9001:h     campaign 9001, hourly
v1:fc:{u:42}:r:55501:d    creative 55501, daily
v1:fc:{u:42}:p:14:d       daypart 14:00, daily

Three details in the implementation carry the lessons:

Reads are batched, writes are deferred. The bid path reads all of a user’s relevant counters in one Redis MGET (possible only because every key shares the {u:42} hash tag — Part 1’s co-location decision paying rent). A missing key decodes as zero: “never shown” and “counter expired” are deliberately the same state. Writes — the increments after an impression — never touch the bid path at all: they flow through a bounded queue and land in Redis asynchronously via a Lua script that increments and sets the window’s TTL atomically, so a counter and its expiry can’t get separated by a crash between two commands.

The values are ASCII integers, not binary. Part 1 made a case for packed binary values; frequency counters do the opposite, and the contrast is the lesson: Redis’s INCR arithmetic only works on its native integer strings. Encoding follows the operations you need, not a house style. Segments are read-only blobs — pack them. Counters are server-side arithmetic — leave them in the server’s format.

Async writes make caps eventually-exact, on purpose. Between an impression and its increment landing, a burst of requests can see the stale count, so a user can briefly exceed a cap. That’s a chosen trade — the alternative is putting a synchronous write on the hot path — and it’s bounded, measured, and disclosed. A cap is a promise about aggregates, not a mutex. (What happened when this stage’s read side went wrong is Part 4’s story, and it’s the best story in the series.)

The auction: ranking, pricing, and the shading gap#

When the surviving candidates reach ranking, the decision is two lines of logic: highest score wins the slot, price breaks ties (bidder-core/src/pipeline/stages/ranking.rs). The bid price itself is the campaign’s catalog CPM, converted from cents at the response boundary — the model’s score decides which campaign bids, not how much.

That simplicity is an honest, disclosed gap, and understanding why requires one piece of auction history. Ad exchanges historically ran second-price auctions — the winner pays the runner-up’s bid plus a tick — under which the optimal strategy is beautifully simple: bid your true value, since overbidding never changes what you pay. Through 2017–2019 the industry moved to first-price — you pay exactly what you bid — completed when Google moved Ad Manager to a unified first-price auction in 2019. Under first price, bidding your true value systematically overpays, so every serious DSP grew a bid shading layer: a model estimating the lowest price that still wins, trained on the win/loss feedback of its own bids.

This bidder bids catalog price, unshaded. In a first-price world that overpays — stated plainly rather than hidden. The roadmap fix is a feedback loop, and the raw material for it already exists in the system: every win notice carries the clearing price. Which is exactly why the win notice deserves the rest of this post.

The win notice is a fraud surface#

Winning a bid ≠ knowing you won. The exchange tells you by calling back a URL — the nurl — that you embedded in your own bid response. That callback is what increments spend counters and frequency counters. Now consider what happens if that endpoint trusts its callers: anyone who can guess its shape can fabricate wins — draining a competitor’s advertiser budgets, or inflating a user’s frequency counters until real ads stop serving to them. The win endpoint is a payments webhook wearing an ad-tech costume, and it gets the same treatment:

The design, piece by piece (bidder-server/src/win_notice.rs):

• The signed message is the notice’s identity: request_id|imp_id|campaign_id|creative_id, HMAC-SHA256 with per-exchange secrets (falling back to a default). An attacker can’t produce a valid token for a bid they never saw.
• Verification is constant-time. A naive byte-by-byte comparison leaks, through response timing, how many leading bytes of a guessed signature were right — turning a 2^256 search into a linear one. The comparison uses a constant-time equality primitive. This costs one line and is non-negotiable for any MAC check.
• Replays are deduplicated with SET NX on the (request, imp) pair, with a TTL — the same valid notice delivered twice (retries happen!) counts money once.
• The failure mode fails closed. If Redis errors during the dedup write, the notice is treated as a duplicate, not as new. The system would rather occasionally miss counting a real win than ever double-count one. Pick your failure direction for every money path, on purpose, before the failure happens.

What to take away#

1. Money is integer cents, always. And every money path needs a chosen failure direction — the pacer fails toward not spending; the win endpoint fails toward not counting.
2. Reserve-and-release is the shape of concurrent spending. Reserve atomically before you commit, release for every loser after ranking — or leak budget invisibly.
3. Enforcement and smoothing are different layers. A counter stops overspend; pacing to a traffic forecast is a policy on top (throttling, in most published systems).
4. A frequency cap is a schema — entity × window, independent counters, TTL set atomically with the first increment, missing = zero, and encoding chosen by the operations you need.
5. Any callback that moves money is a fraud surface. Sign identity, verify in constant time, deduplicate replays, fail closed — and know exactly what the signature does not cover.

Part 3 climbs into the scoring stage — serving an actual ML model inside a 4ms budget, and the boot-time check that keeps the model you serve honest with the model you trained.

References & further reading#

• Budget pacing for targeted online advertisements at LinkedIn — KDD 2014, the canonical throttling design
• Smart pacing for effective online ad campaign optimization — KDD 2015, feedback-controlled pacing that drops the least valuable auctions first
• First-price auctions for Google Ad Manager — the announcement that moved the industry off second-price
• OpenRTB specification — nurl, the price macro, and auction mechanics live here

Disclaimer: I don’t work in ad tech. This series documents a personal project built to learn high-performance systems engineering, using real-time bidding as the problem domain because its constraints (hard latency SLAs, large working sets, high request rates) are unusually honest teachers. Industry references are from public engineering material; my own numbers come from synthetic workloads on developer hardware and are reported with their caveats. Corrections from people who build these systems for a living are very welcome.