LLM Evals: How to Test LLM Apps and Agents

A user complains that the support bot’s refund answers are too curt. I tweak the prompt — “be warmer, acknowledge the customer’s frustration.” I try three questions in the playground. All three answers look great. Ship it.

A week later a different complaint arrives: the bot has started approving refunds it should refuse. The warmth instruction nudged it toward agreeableness, agreeableness leaked into policy decisions, and none of my three playground questions happened to test a refusal. Nothing crashed. No test failed — there were no tests. I changed the system and had no way of knowing what else I’d changed.

The everyday version of this: my uncle taught me to drive, watched me park once, and declared me ready. The driving authority disagreed — they have an examiner with a rubric: a fixed route, specific maneuvers, defined mistakes, pass or fail per item. Not because the examiner is smarter than my uncle, but because “seemed fine to me” doesn’t survive a thousand drivers. An eval is that rubric for an AI system. My playground check was the uncle.

Why the tests I already write don’t transfer#

Backend instinct says: write a test.

def test_refund_answer():
    answer = support_bot("Customer wants a refund, bought 12 days ago")
    assert answer == "Refund approved per policy."   # fails on run 2

• The assert dies immediately: the same input produces differently worded output on every run. There is no single right answer to compare against.
• Loosening it to assert "refund" in answer passes for the wrong reasons — “I cannot process refunds” contains the substring.
• And the deeper problem: for “is this answer good?”, correctness isn’t a string equality — it’s a judgment. Something has to make that judgment, and that something can itself be wrong.

So the three pillars of ordinary testing — deterministic output, a known expected value, a trustworthy comparator — all fall over at once, and they fall for one reason: an LLM app’s behavior is a distribution, not a function. No editor shows a diff of a distribution, so every prompt tweak, model upgrade, and retrieval change reshuffles behavior everywhere at once, silently.

The fix is not to give up on testing. It’s to rebuild it one level up.

Which is to say: write the rubric down. A fixed route of real cases, driven several times because one clean lap proves nothing, marked by the cheapest marker that can catch the mistake — and the marks are there to be read, not totalled.

Anatomy of a harness#

Four parts, and every eval tool on the market is some skin over exactly these:

flowchart LR
  P["production failures"] -->|"one new case each"| G["golden set"]
  G -->|"case.input"| R["runner: N trials per case"]
  R -->|"N outputs per case"| GR["grader"]
  GR -->|"pass / fail + reason"| REP["report"]
  B["baseline (last run)"] -->|"per-case diff"| REP

The golden set is a frozen collection of real scenarios — not invented ones. The honest way to build it is to read your own traces (the practitioner consensus is depressingly consistent: read 100+ real interactions, tag what actually went wrong in each, and the failure categories that emerge are your first golden set). Start small — ten to thirty cases beats zero for months — and grow it by one case per new production failure, the same way a bug becomes a regression test.

The runner runs every case N times, because a single run of a nondeterministic system is an anecdote. The grader decides pass/fail — the entire next section. The report does the one thing that makes any of this worth it: diff against the last run, so a change that fixes three cases and breaks two shows up as exactly that, not as “score went up a bit.”

Everything below is from my llm-evals-lab repo — stdlib-only Python, runnable offline against a deliberately flaky fake app:

@dataclass(frozen=True)
class EvalCase:
    id: str
    input: str
    expected: str | None = None     # reference answer, if one exists
    tags: tuple[str, ...] = ()      # slice results: "refunds", "multi-hop"

class Runner:
    def __init__(self, app, grader, trials: int = 3): ...

    def run(self, cases):
        for case in cases:
            for _ in range(self.trials):
                response = self.app(case.input)          # the app is just a callable
                grade = self.grader.grade(case, response.output)
                # ...record output, grade, tokens, latency per trial

• EvalCase is frozen on purpose — a golden set that quietly mutates can’t be diffed across weeks.
• tags are what turn a score into a diagnosis: “82% overall” says nothing; “97% on lookups, 40% on refusals” says exactly where to dig.
• The app under test is any callable — a bare prompt, a RAG pipeline, a full agent. The harness doesn’t care, which is the point: the harness outlives every framework choice inside it.
• Tokens and latency are recorded per trial, first-class. A fix that doubles cost is not a fix; almost no public benchmark scores this, which is exactly why my own harness must.

The grader ladder — spend judgment like money#

Grading has a ladder, and the rule I’ve adopted: climb only as high as the failure forces you to.

The under-appreciated rung is the first one. A shocking share of real failures are mechanical: the output wasn’t valid JSON, the required policy citation is missing, the refusal didn’t happen. From the lab:

class JsonSchemaGrader:
    def grade(self, case: EvalCase, output: str) -> Grade:
        try:
            data = json.loads(output)
        except json.JSONDecodeError as e:
            return Grade(False, 0.0, f"not valid JSON: {e}")
        missing = [k for k in self.required_keys if k not in data]
        return Grade(passed=not missing,
                     score=1 - len(missing) / max(1, len(self.required_keys)),
                     reason=f"missing keys: {missing}" if missing else "schema satisfied")

• Every grade carries a reason. The score aggregates; the reasons are what you actually read. A report that says “62%” is a mood; a report that says “9 of 11 failures are missing keys: ['policy_id']” is a task.
• Partial credit (score vs passed) exists because “output had 4 of 5 required fields” and “output was HTML” should not be the same zero.
• One deliberate dogma, borrowed from the practitioners I trust on this: verdicts are binary. Pass or fail, never a 1-to-5 scale — nobody agrees on what a 3 means, and models drift toward flattering middles. If a case feels like a 3, that’s two different cases wearing one id.

The judge is a witness, not an oracle#

Sooner or later a case needs judgment — “is this explanation actually correct?” — and the affordable judge is another LLM. This works, measurably, but only if you treat the judge as a biased witness whose biases are documented:

• Position bias: in pairwise comparisons, judges favor one slot. Measured on MT-Bench: GPT-3.5 leaned on position in ~50% of judgments, Claude-v1 in ~70%.
• Verbosity bias: both judges preferred the longer answer more than 90% of the time — even when the extra length added nothing.
• Self-enhancement: models grade their own outputs kindly — GPT-4 favored itself by ~10%, Claude-v1 by ~25%.

The mechanical fix for position bias is beautiful in its paranoia — ask twice, swapped, and only count a verdict both orderings agree on:

sequenceDiagram
  participant H as Harness
  participant J as Judge model
  H->>J: which is better? slot A = ours, slot B = baseline
  J-->>H: WINNER: A
  H->>J: same answers, swapped. slot A = baseline, slot B = ours
  J-->>H: WINNER: A
  Note over H,J: slot A won both times. That is a position preference, not a verdict. Score it a tie.

def pairwise_judge(client, case_input, answer_1, answer_2) -> str:
    first = _one_round(client, case_input, answer_1, answer_2)   # answer_1 in slot A
    second = _one_round(client, case_input, answer_2, answer_1)  # swapped
    if first == "A" and second == "B":
        return "answer_1"        # won from both seats
    if first == "B" and second == "A":
        return "answer_2"
    return "tie"                 # disagreement = the judge answered from position

• Two calls instead of one — position de-biasing literally doubles judge cost, which is another reason to keep cases low on the grader ladder.
• The rubric prompt (in the repo) forces reasoning before verdict — a judge that states the verdict first just rationalizes it afterwards.
• The judge gets a way out: an unparseable or hedged response fails loudly as a harness error instead of silently passing.

And the rule above all judge rules: never trust an unmeasured judge. Before the judge grades anything unsupervised, it grades ~30 outputs a human already labeled, and two numbers come back: agreement, and the dangerous one — false-accept rate, how often the judge passes what the human failed. A lenient judge inflates every future score by roughly that rate; “we improved four points” can be zero real improvement. The good news from people who’ve done this seriously: with a few iterations on the judge prompt (few-shot examples of the human’s critiques), 90%+ agreement is reachable — and the ceiling is real anyway, since even two humans only agree with each other about 81% of the time on this kind of grading.

report = calibrate(judge, labeled_outputs)   # ~30 human-labeled cases
# report.agreement          -> 0.87
# report.false_accept_rate  -> 0.08   the metric-inflating direction
# report.trustworthy()      -> True   only now does the judge grade alone

Reliability is a different number than capability#

This is the most important idea I took from the agent-benchmark papers, and it fits in one line of math. If an agent succeeds a task with probability $p$ per attempt, then over $k$ independent attempts:

pass@k asks did it EVER succeed? — the demo number, and it only goes up with retries. pass^k asks did it succeed EVERY time? — the production number, because a customer-facing agent doesn’t get to fail colorfully three times out of eight. Same agent, same $p$, wildly different stories:

The real-world version of that chart, from the tau-bench paper (the customer-service agent benchmark that introduced pass^k): GPT-4o scored ~61% per attempt on retail tasks — and about 25% at pass^8. Same model, same tasks; one number is a press release, the other is whether you’d let it near customers. This is why the lab’s runner does N trials per case and the report prints both numbers:

- cases: 3, trials per case: 5
- pass rate: 66.7%  (95% CI 40.0% - 100.0% -- mind the width)
- pass^3 (all 3 trials pass): 36.7%
- cost: $0.0096 across 15 trials

That confidence-interval line is the other statistics lesson, and it’s brutal: on a tiny golden set the uncertainty swallows the score. A 4-point improvement measured on 40 cases is noise wearing a suit. The lab computes the interval by bootstrap — resample your cases with replacement a couple of thousand times and look at the spread; no distribution theory needed, and it makes small-set overconfidence impossible to sustain.

For agents specifically, one more split matters: grade the outcome (did the database end in the right state — tau-bench literally diffs final state against a goal state) separately from the trajectory (which tools were called, in what order, at what cost). The two disagree in instructive ways — right answer via a forbidden path, or a “failure” where the agent found a legitimately better route than the reference solution. When outcome and trajectory disagree, the case is trying to tell you your spec is incomplete.

The eval itself can be the bug#

The story that made me respect this field: Anthropic ran Claude Opus 4.5 on CORE-Bench (a benchmark of reproducing computational-research results) and got 42%. After fixing the eval — graders that rejected “96.12” because the reference said “96.124991”, ambiguous task specs, stochastic tasks with no stable answer — the same model scored 95%. More than half the measured failure was the eval’s own bugs.

So graders get debugged like any other code, and the discipline that keeps a suite honest over time is splitting it in two:

• Capability evals — things the system can’t do yet. Expected to start near 0%. This is eval-driven development: write the eval for the feature before building the feature, watch the score climb as you build.
• Regression evals — things that must never break. Held at ~100%; any dip blocks the change. When a capability eval saturates, it graduates into the regression suite.

One suite tells you where you’re going; the other guarantees you don’t pay for progress with breakage. My playground-tested refund bot from the opening failed precisely because it had neither.

When evals mislead#

The limits, honestly, because a harness can lie in both directions:

• Goodhart’s law comes for golden sets. Optimize against the same 60 cases for months and you’re fitting to them, not to users. Rotate cases in from fresh production failures; retire the memorized.
• Offline evals can’t see drift. The world changes under a frozen dataset — user behavior, upstream models, data distributions. The online layer (canary deploys, A/B tests, production monitoring) is a separate discipline this post doesn’t cover, and no offline score substitutes for it.
• The judge drifts too. Judge model upgrades silently change grading standards — pin judge versions, and re-run calibration when anything about the judge changes.
• Public benchmark numbers are marketing until reproduced. The agent-memory vendors currently publish mutually contradictory scores on the same benchmark — grading your own homework is the norm, not the exception. The only benchmark numbers I fully trust are ones from a harness I can run.

Every piece of this has a backend twin I already trusted: a golden set is a regression suite; N-trials-per-case is what you’d do for any flaky test; baseline diffs are snapshot testing; canary deploys are online evals with a blast radius. The tooling ecosystem (pytest + promptfoo, LangSmith, Braintrust, OpenAI’s evals) packages these same four parts — the reason to build a tiny one from scratch first is the same reason to implement a hash map once: so the tools stop being magic.

What survives from all of that#

• LLM app behavior is a distribution; testing it means freezing cases, sampling several runs, and grading the samples.
• Grade with the cheapest grader that catches the failure; save the judge for what only judgment can grade.
• A judge is a biased witness: position-swap pairwise comparisons, reasoning before verdict, and never unleash a judge whose false-accept rate you haven’t measured.
• Report pass^k next to pass@k — reliability and capability are different numbers, and production pays for the first one.
• Put a confidence interval on everything; small golden sets have wide ones.
• Debug the grader with the same seriousness as the app — half of a bad score can be the eval’s fault.
• Split capability evals from regression evals; graduate cases from one to the other.

The question I’m left with, and the one the whole field seems to be leaving on the table: almost no public agent benchmark scores cost — pass rates are printed with no dollars attached, as if a 3% win at 10x the tokens were progress. A pass-rate-per-dollar column in the lab’s report took eleven lines. Why is nobody publishing that number?