Building a Real-Time Ad Bidder in Rust — Part 3: Serving an ML Model in 4 Milliseconds

The scoring stage answers the auction’s central question — which eligible campaign is this impression worth the most to? — with a model that predicts click-through probability per candidate. When the ML scorer is enabled, its budget is a 5ms envelope — 4ms for inference itself, 1ms for packing features in and scores out — inside a pipeline that must finish in 40. This part is about everything around the model: where it runs, how it fails, how you know it’s computing the same numbers it was trained to compute, and how a new one rolls out without betting the fleet on it. Serving ML under a hard latency budget is its own discipline, and almost none of it is about the model.

---

Where the model runs: in-process, and why#

The reflexive architecture is a model server — a separate service behind an RPC. It’s the right call surprisingly often, and the wrong call here:

	In-process (chosen)	Model server
Latency	Function call; microseconds	Network hop: serialization + RTT + its own queueing — a second tail to manage
Deploys	Model ships with (or hot-loads into) the bidder	Model deploys independently
Scaling	Scales with the bidder, like it or not	Scales independently; GPU pooling possible
Memory	Every pod carries the model	Once per server pool
Blast radius	Model bug lives inside the bidder process	Isolated behind an API

The tiebreaker is arithmetic: a 4ms inference budget minus an RPC’s p99 leaves nearly nothing for actual inference. For a small pCTR model — thousands of parameters, not billions — the model is cheaper than the network hop to reach it. So the ONNX model runs inside the bidder process via ONNX Runtime (the ort crate). The moment models grow past what every pod can carry, or a second consumer needs the same predictions, the answer flips — that’s not a failure of this design, it’s the boundary of it.

The serving mechanics#

Three mechanisms make in-process serving production-shaped (bidder-core/src/scoring/ml.rs):

A session pool, because inference wants exclusive access. ONNX Runtime’s Session::run requires &mut self — it caches I/O binding state internally. The pattern is a Vec<Mutex<Session>> with an atomic round-robin counter: the pool size is the parallel-inference cap, and lock contention only appears when concurrent scoring exceeds it.

Batching, padded to a fixed rhythm. All of a request’s candidates are scored in one call — an N×13 feature matrix — capped at a max batch, and padded with zero rows up to a multiple of 8. The padding looks odd until you know the reason: models exported with a fixed batch axis recompile per new shape, so feeding the runtime only a few distinct shapes avoids that entirely, and for dynamic-axis models the padding is a no-op. Padded rows’ scores are discarded.

Hot reload by atomic swap. A file watcher polls the model path every 5 seconds; on change, it builds a complete new session pool off to the side and swaps it in atomically — the same ArcSwap snapshot pattern as Part 1’s catalog, because it’s the same problem: replace shared immutable state under readers without ever blocking them. In-flight requests finish on the old model; the next request gets the new one.

That watcher hides a small war story. The file-watching crate’s recommended backend (native OS events — inotify, fsevents) crashed with a mutex lock failed abort during test teardown on macOS: native background threads and process shutdown disagreeing about lifetime. The fix was the unfashionable option — a polling watcher with no native threads at all. A 5-second reload latency is irrelevant for a model that changes at most daily, and “boring, portable, shuts down cleanly” beats “instant” everywhere it matters. Choosing the less clever mechanism because its failure modes are simpler is a production skill.

The parity check: trust nothing at boot#

The most common way ML systems rot in production is training/serving skew: the training pipeline and the serving path each compute “the same” features, drift apart by one timezone or one default value, and scores quietly shift with no error anywhere — the model is fine, the features lie. (The classic catalog of these failure modes is Google’s Hidden Technical Debt in Machine Learning Systems.) Think: training computes is_business_hours in UTC, the bidder computes it in local time. Every score is now wrong by a factor nobody will ever see on a dashboard.

This bidder’s defense is a boot-time parity check. Alongside every model file ships a small JSONL of {input features, expected score} pairs, generated by the same code that produced the model. Before the scorer accepts traffic, it runs every pair through the freshly loaded session — and refuses to start if any output drifts more than 1e-4 from expected.

When inference fails: all or nothing#

Behind the Scorer trait sits a fallback chain: if Session::run errors, the scorer delegates the entire batch to a plain feature-weighted linear scorer and increments a named counter. The subtle part is the “entire”: if earlier chunks of the batch scored successfully before the failure, those partial scores are wiped first. Half-scored candidates would send the ranking stage a mix of real scores and zeros — silently corrupted winners, no error surfaced anywhere. Degraded-but-consistent beats accurate-but-partial, and the wipe is what makes the fallback safe rather than merely available.

The same thinking runs through the trait boundary itself: Scorer::score_all makes no promise about preserving candidate order or count (a cascade may sort and truncate — see below), so results are never matched back by index. Assumptions a trait doesn’t state are assumptions someone’s implementation will eventually violate.

Rollout: cascades and A/B, as configuration#

New models don’t deserve the whole fleet on day one. Two composable decorators, wired entirely from config, give the standard rollout shapes:

• The cascade is the latency-budget trick used across the industry: a cheap model ranks everything, the expensive model re-scores only the top K. Inference cost stops scaling with eligibility breadth and starts scaling with K — the same “bound the expensive stage” instinct as Part 1’s candidate limit.
• The A/B splitter buckets by a deterministic hash of the user id, so a user always lands in the same arm and the experiment measures a stable population. Two details worth stealing: the hash is seeded per experiment, so concurrent experiments don’t entangle (the same user can be treatment in one and control in another, independently); and the hash is a fast non-cryptographic one — assignment isn’t security-sensitive, so paying for cryptographic hashing would be superstition.
• The config grammar bounds the nesting. The builder accepts ab_test(control, treatment) where each arm is a leaf or one cascade — and nothing deeper. A cascade-of-cascades is unrepresentable in valid config. The same principle as Part 1’s pub(crate) bitmaps: when a structure shouldn’t exist, make it inexpressible, not discouraged.

What’s honestly synthetic here#

Stated plainly, because it’s the boundary between what this project demonstrates and what it doesn’t: the model itself is a hand-built test fixture — a 13-feature logistic model, $\sigma(W \cdot x + b)$, with fixed weights, generated by a small tool that emits the ONNX file and its parity JSONL in one command. It exists to prove the serving machinery, not to predict clicks. One feature is a stable placeholder. There is no training pipeline and no calibration — and calibration matters in this domain: a bidder prices with pCTR, so a model that ranks well but runs systematically hot loses real money, not just ranking quality.

Building the real thing is a known path rather than a mystery: public CTR datasets exist at serious scale (Criteo’s 1TB click logs is the standard), and the win notices from Part 2 are exactly where a production feedback loop gets its labels. The serving side was the systems lesson; the training side is a different project.

---

What to take away#

1. Under a single-digit-ms budget, the network hop costs more than the model. In-process serving wins for small models — and know exactly where that boundary is.
2. Make artifacts carry their own acceptance tests. A model that ships with input/expected pairs, verified at boot with a refuse-to-start posture, turns silent skew into a loud startup failure.
3. Fallbacks must be all-or-nothing. Wipe partial results before delegating; degraded-but-consistent beats accurate-but-partial.
4. Roll out with structure, not courage. Deterministic per-user A/B arms, cascades that bound expensive inference to a shortlist, and a config grammar that makes bad shapes unrepresentable.
5. Prefer boring mechanisms with simple failure modes — a polling file watcher over native event APIs is slower and better.

Part 4 is the reckoning: everything the series has claimed so far, pushed until it broke — circuit breakers and backpressure under real failure, and the load tests that found the one stage quietly eating three-quarters of the pipeline.

References & further reading#

• Hidden Technical Debt in Machine Learning Systems — the canonical catalog of ways ML systems rot, training/serving skew included
• ONNX Runtime — the inference runtime embedded in the bidder
• Criteo 1TB click logs — the public dataset a real pCTR model would train on

---

Disclaimer: I don’t work in ad tech. This series documents a personal project built to learn high-performance systems engineering, using real-time bidding as the problem domain because its constraints (hard latency SLAs, large working sets, high request rates) are unusually honest teachers. Industry references are from public engineering material; my own numbers come from synthetic workloads on developer hardware and are reported with their caveats. Corrections from people who build these systems for a living are very welcome.