Eval-driven development: how to choose model, retrieval, and design by testing instead of guessing
文章目录
This piece is for engineers who must pick a model, retrieval setup, and agent design from day one and need those picks to stay grounded, and it also helps leads and managers who weigh cost against quality, with counts and rates taken from published team reports and papers to keep each claim grounded. The core move: define what good means as evals before you build, then every later question becomes testing.
Quick grasp
What it is. Eval-driven development means writing the evals before you build the agent. An eval is a scored test of behavior. You give the agent an input, watch what it does, and score if that was acceptable. A golden pairs an input with a picture of good behavior. It says what must happen, what must not happen, and what good looks like. A metric is the check you score, as a rule in code or a rating from a model. A judge rates what rules cannot see, such as tone. A trace is the full record of steps and tool calls. Once these evals exist, model choice, retrieval tuning, and architecture stops being debate and becomes testing.
Why it exists. Every AI project starts with open design questions: which model is enough, which retrieval setup helps, single agent or router, which tools. Without evals up front, each choice is argued on opinion and the loudest demo wins. Agents then fail quietly, because a small prompt edit or model swap shifts behavior elsewhere and no test catches it. Evals defined in advance fix what good means, so each design option must prove itself on the same cases.
How it works. Define good first, then test every choice against it. You write the failing cases before the agent passes them. You fix. You keep the case so the bug cannot return unseen.
-
Define good up front as goldens with clear metrics, before the agent passes them, so model, retrieval, and design options face the same check from the start.
-
Fix the agent until the new case passes, using code checks where a rule can decide and a judge only where reading is needed.
-
Keep the full trace for each run, so you can see which step broke when a case fails.
-
Test each design option on the same set and ship only repeat wins, growing the set with each bug users find, so the set tracks real use over time.
When to use it. Use it from the first design decision — model, retrieval setup, tools, architecture — and keep it for every later change; when a prompt edit, model swap, or new tool can shift untouched answers; when tone or safety bounds matter beyond string match. Skip it only for fixed logic with fixed outputs, where unit tests already hold.
Important: A single pass can mislead because agents vary across runs. What it means for us: We score rates across repeats and guard known failures with saved cases. Why it matters: Ship choices then rest on proof, not on how the last demo looked.
Where this goes
-
Level 1: why normal tests fail for agents and why rates must replace single matches.
-
Level 2: how to build the right test set from real failures and keep a held-out guard set.
-
Level 3: what only a model judge should check and what code should check instead.
-
Level 4: where people fit as definers of good and auditors of the judge.
-
Level 5: whether to check the end result or the steps based on control need.
-
Level 6: how to read noisy scores and ship only on repeat wins without key breaks.
-
Level 7: how evals guide choice of model, retrieval, tools, and agent shape.
Keep a single thought from this start: do not trust how the agent answered last time. Write that case down, score it repeatably, run it after the next change, keep it. That loop is eval-driven development. The rest shows how to keep that loop light, using lessons from a docs support agent, an IT support agent with strict steps, and a health coach where wording carries risk.
Why one check is not enough for agents
This level answers: Why does checking an agent once tell you so little?
You try the agent by hand. You type one message. The answer looks good. You ship it. Then it fails for real users.
This happens to almost every team. Hand-checking feels fast. It also lies. One try shows one path. The agent has many.
Normal software gives the same output for the same input. This is called deterministic. Deterministic means fixed and repeatable. You put in two plus two. You always get four. You run the test once. The result stands.
Agents are not deterministic. The same input can give a different output each time. Two reasons cause this. Both matter.
The first reason is sampling randomness. Sampling means the model picks from likely next words, not always the top one. This keeps language natural. It also adds variation: the same prompt can give different wording or a different choice each run.
You can turn down this randomness with a setting often called temperature. Low means safer and more repeatable. High means more varied. The limit is simple. Low reduces variation without removing it, and very low settings can make answers stiff. Most live agents keep some randomness, so expect change run to run.
The second reason is many valid phrasings. Many different answers can all be correct. A user says the laptop will not connect to the VPN. One good reply says the ticket is open and help will come soon. Another says the same thing in different words. A user accepts both.
Exact-match checking fails here. Exact match compares the output to one fixed string. Different words fail the test even when the meaning is right. That test is too strict. It punishes correct variety.
There is a third problem. The agent chooses its own steps. A normal function follows one fixed path. An agent picks tools at run time: which tool, in which order, whether to retry.
Take a small example. The agent has two tools: search for help articles, and ticket for opening an IT ticket. The user writes that the VPN will not connect.
Now watch three runs with the same input. First run: it searches, finds a fix, sends steps, opens no ticket. Second run: it opens a ticket at once, never searches. Third run: it searches, finds nothing useful, then opens a ticket.
Which run is correct. It depends on your rule. If the rule says always try help articles first, the second run broke it. If the rule says VPN failures must always become tickets, the first run broke it. If both paths are allowed, all three may be fine.
The point is clear. You cannot check one fixed path. The agent is meant to choose. The same input can lead to different tool orders, and more than one order can be right. A classic test that expects one call order breaks by design.
So change the question. Do not ask if it passed this once. Ask how often it passes. This needs two plain terms.
A trial is one run of one case. You give the input, let the agent act, score what happened. That is one trial.
Pass rate is how many trials passed out of how many ran. Run the same VPN case five times: three follow your rule, two skip the search step. The pass rate is three out of five, which is sixty percent. That number tells a story. Sixty percent means shaky, skipping search about half the time. One lucky run would have said all good or all bad. Five runs show the truth.
Important: One run of an agent proves almost nothing because wording, choices, and tool paths change each time. What it means for us: We score the same case several times and read the share of passes, not a single yes or no. Why it matters: Rates reveal flaky behavior early, before users feel it.
How many repeats do you need. Start small. Grow with risk and doubt.
Start with three trials for a new or everyday case. Three is cheap and fast, and tells far more than one. If all three agree, that is a clear signal for most low-risk cases: all pass means likely solid, all fail means likely broken. The limit is that a fault showing once in ten runs will often hide in three. So three is a first filter, not proof.
If the three trials disagree, treat that case as unstable. Unstable means the result flips run to run. Do not average it away. Run it more times to get a real rate, and keep it as a key case. Flaky cases break first when wording changes or models swap. The limit is cost, so spend repeats here, where they pay off.
Give critical cases more repeats from the start. Critical means failure hurts a lot: wrong routing annoys, while deleting data or skipping a safety check is serious. For serious cases, more trials up front give even a rare miss a chance to show. More repeats lower the chance of missing a rare fault but never bring it to zero. Many clean trials mean strong evidence, not perfect safety.
Hand-checking still has a place for fast feel, demos, and big breaks. Its limit is that it cannot measure stability: one sample hides the spread.
The shift is small but deep. Old thinking treats a test as a gate that passes or fails once. New thinking treats it as a measurement that passes at some rate. From did it pass to how often does it pass. That shift makes agent work testable, and all later steps build on it.
Carry forward
-
Single manual tries hide variation, so judge agents by repeat behavior rather than one lucky output
-
Same input can yield different valid wording and different tool paths, so checks must allow more than one right answer
-
Repeating cases and reading pass rates turns flaky behavior into visible signal the team can act on
How to build a test set that stays honest
This level answers: How do we choose agent tests that catch real breaks?
Many teams ask for a fixed count. They want a rule for how many tests make evals real. There is no such rule. Count alone does not make a set good. A large set of clean imagined cases can still miss real faults. A small set drawn from real pain can catch a lot. What matters is method, not number.
The method has five parts. Seed from real failures. Cover what must happen, what must not happen, and what must keep working. Grow the set with every bug that reaches users. Hold back a hidden set. Use generated cases with care.
Start from real failures, not imagined happy paths. Happy paths picture a polite user with a clear request. Real users write in a rush, with typos, half sentences, two asks in one message, or anger when something breaks. Neat test inputs describe a world that does not exist.
Seed from reality instead, using three sources. First, past support tickets and chat logs: take recent messages, strip private details, keep the messy wording. Second, bad live runs: every time a demo goes wrong, save that input. Third, bugs from manual checks: if the agent failed once, add that case before you fix it.
This works because builders write differently from users. Real inputs carry slang and edge forms your mind skips, while logs keep them. Early logs are thin and new features have none yet, so the first set will still have gaps.
Next, cover three kinds of behavior for each job. A job means one task the agent must handle. For each job, test what must happen, what must not happen, and what must keep working.
What must happen covers the core duty. Take a help agent that just gained a ticket tool: when a user reports a fault that needs staff action, it must open a ticket with the right details. The limit of testing only this kind is false comfort. The agent can gain the new duty yet break old ones.
What must not happen covers the boundary where the agent must hold back. The same agent must not open tickets for how-to questions, such as how to reset the VPN, and should answer from docs instead. Without must-not tests, that drift ships unseen.
What must keep working covers old behavior from before your latest change. Before the ticket tool, search worked fine. Lock that in with tests and run the full set after each change.
Grow the set with every bug that reaches users. Escaped bug means a fault users felt before tests caught it. When that happens, add that exact case and keep old passing cases as guards.
This mirrors classic software work: each escape earns a lasting test. A set that only grows turns slow, so review at times, merge near copies, and keep the odd wording that caused the break.
Judge enough by falling surprise, not by count. If each release still brings a brand-new kind of mistake, the set is too thin.
Hold back a hidden set you never tune on. Tuning means editing prompts or code until tests pass. If you tune on all cases, scores climb yet live use still breaks.
That trap is called overfitting: the fix memorized your tests instead of learning the real task, like a student who memorizes past exam answers yet blanks on fresh questions.
Important: Tuning on every test can inflate scores without fixing the real task. What it means for us: We keep a small group of cases aside and never edit based on single hidden cases, only on the broad score. Why it matters: A gap between tuned scores and hidden scores warns us that gains will not survive fresh user wording.
The defense is simple. Keep a small group of cases aside and iterate freely on the main tuning set, then run once on the hidden set before declaring victory. If tuning scores are high but hidden scores are low, you overfitted. Refresh the hidden set from fresh traffic. Do not fix single hidden cases by hand, or it stops being hidden.
Use generated cases with care. Generated means made by a model rather than drawn from users. When real traffic is thin, fakes help you start.
But treat fakes as a scaffold, a short-term frame you later replace. Made-up inputs tend to be cleaner than real ones. An agent that aces clean fakes can still fail on messy reality.
Label quality is another limit: a made-up case carries the maker's guess about the right answer. If that guess is wrong, the test bakes in the error. So review made-up cases by hand and favor real cases as they arrive.
In one line, the method is this. Seed from real messages and past faults. Cover each job three ways. Add each escape as a new guard. Judge enough by falling surprise, not by count. Hold back a hidden slice to catch memorization. Use fakes to start, then let live use lead.
Carry forward
-
Honest sets start from real user wording and past faults rather than neat imagined prompts
-
Each job needs guards for required action, for limits that must hold, and for old behavior that must not regress
-
Escaped faults become lasting tests, while a hidden slice and careful use of made-up cases keep scores truthful
Level 3: Use code first, a model judge for taste, people for direction
This level answers: which checker should check what — a program, another AI model, or a person?
Many teams use the wrong checker. They ask a model to count things a program could count for free. Or they ask people to re-read answers daily, which is slow. The fix is simple: a program first, a model judge second, people third.
Three plain definitions. A code check is a small program that answers yes or no: is the order number right, does the output parse? A model judge is another model that scores the reply against a short rubric with examples. A trace is the full log of one run: request, tool calls with inputs, final answer.
Code first for everything a rule can check
Code checks are fast, cheap, and repeatable. Run them on every change. If you can write the check as a rule, write a program, not a model prompt.
Four code checks cover most basics. The right tool was called: for "where is my order," the log should show an order lookup, not a password reset. The arguments were right: the tool can be correct while inputs are wrong, such as a wrong order number from earlier chat. The side effect is real: if the agent says a ticket was created, the program confirms the record, because a confident sentence without one is a failure. A citation marker is present where answers should rest on retrieved material.
Numbers make these checks concrete:
-
47 out of 50 runs called the right tool. That means routing mostly works and 3 failures point to specific confusions.
-
42 out of 50 answers included a citation marker. That means grounding is weak and needs work first.
The limit of code checks: they verify structure and presence, but cannot tell whether wording is kind, clear, or safe. That is the next layer.
Important: Never pay a model to check what a program can check exactly. What it means for us: Move tool, argument, side-effect, and citation checks into programs on every change. Why it matters: It cuts cost and noise, and saves the model judge for reading-level judgment only.
Model judge only for things you must read to score
Some qualities have many right answers no rule can list. Tone, policy nuance, and clarity need a reader.
Tone means the reply fits the setting: factually right but too cold or too long for chat fails here. Policy nuance means holding a fine line: a wellness coach can share general sleep habits but must not diagnose or name a dose, and a good refusal says no warmly with a next step. Clarity means the reply states the rule and applies it to this person across many valid wordings. A program comparing exact words fails them all; a judge reading for meaning passes the good ones.
Rates with a plain reading work here too:
-
36 out of 50 replies passed a tone rubric. That means about a third need shortening or warming up.
-
18 out of 20 borderline replies held the policy line and the redirect. That means behavior is close and 2 misses need new rubric examples.
The limit of a model judge: noisy, biased, and costly on every run. Keep it narrow — taste, nuance, clarity. Where you can point the checker at evidence, such as whether each claim appears in retrieved passages, prefer a focused grounding check over asking "is this true in the world?"
Expect judge bias and design against it
A model judge is not neutral. Three biases, each with a fix. Position bias favors the answer shown first, so judge both orders and count a win only if it wins both ways. Verbosity bias favors longer answers, so reward brevity with one short good example beside one long bad example. Self-preference favors writing like the judge's own model family, so judge with a different family than the one that wrote the answer.
One habit helps all three: the judge writes its reason before the score, with an escape option such as "not enough evidence" so it never guesses. Test the judge like code, with known-bad conversations it must fail:
- 10 out of 11 known-bad cases were caught. That means the judge is sensitive enough, and 1 miss shows which example to add.
One claim from one team: a small judge missed 4 to 5 known-bad cases that a larger model from a different family caught. Treat this as their result. The lesson: test your judge on your known-bad set, and upgrade it or tighten the rubric if it passes cases it should fail.
People define good and audit the judge
People are too slow to score every run, but only they can say what "good" means. Two jobs, done up front and repeated only when the task changes. First, define good with examples: collect great and terrible replies, note why each passes or fails, turn those notes into the rubric. Second, audit the judge: sample 30 to 50 replies, score the same cases blind, compare. Where scores disagree, fix the rubric with the missing example.
- People and the judge agreed on 41 out of 50 sampled cases. That means agreement is decent and 9 disagreements mark rubric gaps.
Then let the judge run alone until the task changes, agreement drops, or a new failure type appears. The limit is scale and drift: when product or policy changes, old rubrics go stale, so periodic audits keep the two aligned.
Carry forward
-
Code is the floor: right tool, right arguments, real side effect, citation present — programs on every change.
-
A model judge is for reading-level judgment only: tone, careful policy line with redirect, clear meaning across phrasings.
-
Fight the three biases with both orders, paired rubric examples, a different model family, and reason-before-score plus an escape option.
-
People define good with examples and audit the judge on samples, then step back until the task changes or agreement drops.
Level 4: Gate on the result when the path is free, check the steps when the path is required
This level answers: should the test check the final result, the steps the agent took, or both?
Older advice says to check only the final result and never the steps. Half right. When any path to a good answer works, step checks only get in the way. But when order matters for safety, law, money, or trust, the steps are the product, and result-only checks miss real failures.
Two plain definitions. A result check asks about the end state: is the report correct, does the ticket exist? A step check asks about the road: which tools ran in which order, was confirmation asked before an irreversible act? A trace is the full log behind both. Outcome decides whether to ship; trajectory explains why it worked or failed.
Free path: gate on the end state
Some tasks have many valid ways to succeed. A research task wants a correct summary with real sources. A bug fix wants failing tests to pass with nothing else broken.
Forcing exact steps here causes three problems. It blocks valid new solutions, since stronger setups find shorter routes. It makes tests brittle, since tool names change while the answer stays right. And it punishes improvement, since a better but different upgrade fails a rigid test.
So gate release on outcome — correctness, coverage, sourcing, real side effects — with partial credit on multi-part tasks:
- 17 out of 20 research answers were fully sourced. That means sourcing is solid and 3 gaps mark retrieval misses.
The limit of outcome checks: they confirm the destination but hide the road. Fine here, because the road was free to vary.
Important: When any safe path is acceptable, judge the destination, not the road. What it means for us: Make ship or no-ship depend on end-state measures like correctness, coverage, sourcing, and passing tests. Why it matters: Better routes pass, tests stay stable, and upgrades stop looking like failures.
Required path: check the steps because the order is the requirement
Other tasks have steps that cannot be skipped. The order itself is a promise to the user or an outside rule.
Eligibility before action is the clearest case: the agent must verify the record before offering options or creating anything. A perfect-looking ticket for someone never eligible looks fine at outcome level and is wrong. Confirmation before an irreversible act is second: tickets, charges, and deletions wait for a clear yes, and no end-state check can confirm the question was asked. Guardrail order is third: a separate review checks each reply before the user sees it, an unsafe draft goes back for rewrite, repeated failure falls back to a safe message.
Step results read as rates:
-
46 out of 50 runs verified eligibility first. That means the guardrail mostly holds and 4 skips must be fixed before release.
-
43 out of 50 runs asked for confirmation in the right place. That means about one in seven acts too early — a blocker for high-stakes actions.
-
A safety probe set blocked 29 out of 30 unsafe drafts. That means the guardrail works and 1 escape defines the next probe.
Cost belongs in step checks too. A correct answer that called an expensive tool 40 times instead of 3 still fails in production. Counts and token use make waste visible: a median of 12,000 tokens with 4 out of 50 runs above 60,000 means a heavy tail worth capping. Promote a step to a gate only with a named rule behind it — a policy, safety need, cost cap, or trust promise.
Always record the trace, even when you gate on outcome
Whatever you gate on, save everything from every scored run: input, each tool call, final state. Recording is cheap. Re-running a failure is costly and sometimes impossible, since the same agent can behave differently next time.
The trace has one job above all: debugging. When a score drops, the trace shows where: wrong documents, misread record, skipped confirmation, right tool with wrong arguments. Without the log you guess; with it you find the faulty step in minutes.
Its second job: the trace becomes the next test. A failed log with a new mistake, such as quoting a ticket number never created, becomes a known-bad case. Even around 10 such examples make a strong set. Logs can hold personal data, so keep them access-controlled.
Two layers in practice
Combine outcome and trajectory as two layers. Layer one is outcome for release: the ticket exists for an eligible user with confirmation, the reply is safe, the summary is faithful, the code passes tests. Run key cases more than once, compare versions on the same cases, ship only on repeated gain. Layer two is trajectory for debugging plus required steps for control: record traces without gating on them, so a strange-but-correct path still ships, while a few steps become gates: eligibility verified, confirmation asked, safety review ran, budget respected.
-
A new version won on 14 cases and lost on 3 out of 50. That means 3 losses need review — one key break is a regression even when the average looks better.
-
A release gate required 50 out of 50 confirmation checks on irreversible actions. That means the bar is perfection there — one silent action costs too much.
A cadence many teams reuse: cheap program checks on every change, model-graded checks on larger changes or nightly, human sampling on a slower rhythm. Against excess: a gating step check that cannot be tied to a named rule belongs in debugging. Gates stay small, named, and explainable.
Carry forward
-
Free path means gate on the end state: correctness, coverage, sourcing, real side effects — or better routes will fail good work.
-
Required path means check the steps: eligibility before action, confirmation before anything irreversible, guardrail order, cost caps.
-
Always record the full trace so failures are localized in minutes and turned into regression cases.
-
Use outcome for release and trajectory for debugging, promoting only named, explainable steps to gates.
Level 5: One score is a range, not a verdict
This level answers: how do you make a ship or no-ship call when the score moves on its own?
Run the same agent on the same test set twice and you get two different numbers. Nothing changed, yet the score moved, say 82 out of 100 first, then 79.
Many teams conclude evals are useless. They are wrong. Think of a bathroom scale showing 70.1 kg one minute and 70.3 kg the next. Still useful. You just do not panic over 200 grams. Agent scores work the same way. One score means around 80 percent, give or take a few points.
This level is about that shift. Stop asking for the exact number. Ask three better questions: what did the new version fix and break, is the gap bigger than the wobble, and where is the line going over time.
Why the same agent scores differently twice
Three sources of wobble. All normal.
First, sampling randomness. The model picks each next word from likely options, not from one fixed answer. Same input can give different wording or a different document picked, and on a small set one or two flipped cases move the whole average by several points.
Second, small test sets. The math makes small sets jumpy by design:
-
With 20 cases, each case is worth 5 points, so one lucky guess swings the headline by 5 points and looks like a big change when nothing changed.
-
With 100 cases, each case is worth 1 point, so the same lucky guess barely moves the average and the number feels calmer.
-
With 200 cases, each case is worth half a point, so luck mostly evens out and small real gains become visible.
That is why teams start with 20 to 50 real cases to catch big breakage fast, but never declare a 2 point victory on a set that small.
Third, judge variation. A code check does not wobble. A tool was either called or not. An LLM judge wobbles a little, because a borderline answer can pass once and fail once.
The score is an estimate with blur around it. Only act on gaps bigger than the blur.
Never compare two lonely numbers
Never compare version A at 80 percent with version B at 83 percent and call B better. Run old and new on the exact same cases, then read wins and losses case by case:
-
Old version passes 47 out of 60, about 78 percent, and you save exactly which 47 passed.
-
New version passes 50 out of 60, about 83 percent, on the same 60 with the same judge.
-
The flips matter more than the averages: 5 cases went from fail to pass, but 2 went from pass to fail, which is breakage hiding inside a rising average.
A change that fixes 5 and breaks 2 is a trade-off, not a clear win.
The VPN story shows why. A docs agent had search tools, then the team added a ticket filing tool. The rule was plain: a broken thing like my laptop will not connect to VPN should file a ticket, while a how-to question like how do I connect to VPN should search docs and never file a ticket.
They ran the baseline, added the tool, ran the same eval again. The average went up because old ticket failures now passed. Looked ready to ship.
The per-case view told the real story. One how-to case had flipped from docs search to ticket filing: the tool description said use it for VPN issues, so any mention of VPN matched. The fix was one sentence, narrowing it to requests where IT must act.
Gate only on repeated, clear regression
The automatic gate should not fail on one noisy run. Fail only when the new version is clearly worse than the saved baseline twice in a row.
Important: block the release only on a repeated clear regression, never on one noisy run. What it means for us: the candidate must lose badly against the pinned baseline twice in a row on the same cases. Why it matters: a single-run gate blocks good changes and teaches the team to ignore the gate.
In practice:
-
Keep production scores saved as a pinned baseline, for example last release at 48 out of 60 with the same judge and cases.
-
On each change, re-run baseline and candidate on the same cases, so the comparison is fair.
-
Block only if the candidate loses by enough that luck is a weak excuse, for example 4 or more net cases worse on a 60 case set, twice in a row.
Cheap code checks can block every commit because they are fast and stable. Slow judge checks run per change on about 20 cases and nightly on the full set.
A small habit keeps noise down. Run each case 3 times first. If all 3 agree, one run is enough going forward. If they split, mark that case flaky and give it 5 to 10 repeats, because flaky cases hide the next regression. Give critical safety cases 10 trials and keep low-risk style cases at 3.
Watch the line, not the dot
One score is a dot. A dashboard is the line connecting dots across releases. Track three lines together: pass rate, cost per run, and latency.
That chart catches two things no single check can catch. First, bad deals. Quality moving from 47 out of 60 to 50 out of 60 sounds good until you see cost doubling and latency rising from 2 seconds to almost 5 seconds. Second, slow drift. Each change looks fine alone, but over a month the line sags 8 points.
Close the loop with live data. Cheap code checks on every live reply plus sampled judge scores on about 5 percent of threads surface failures that get added back as new cases.
Carry forward
-
A single score means around 80 percent give or take a few points, so compare old and new on the same cases and read per-case flips before trusting a 3 to 5 point average move.
-
Average up with one key case broken is a regression, so tag must-pass cases and block the release when any flips from pass to fail.
-
Gate only on repeated clear loss against a pinned baseline, and track quality plus cost plus latency on one chart so drift and bad deals stay visible.
Level 6: Evals pick the whole system, not just the prompt
This level answers: which model, which retrieval setup, and which architecture should we ship?
Without evals, system arguments run on opinion. One person says we need the biggest model. Another says we need a re-ranker. A third says we need multiple agents. The loudest voice wins, and the system ends up more expensive and complex than needed.
With evals, the same argument takes an afternoon. Run option A and option B on the same cases with the same judge, then read which cases got fixed, which broke, and what it cost.
Decision 1: which model is enough
The default mistake is using the strongest model for everything. Feels safe, but slow and expensive, and often unnecessary. Run two models on the same cases with the same prompt, tools, and judge, and compare per-case results.
Three patterns show up again and again:
-
Simple tasks where small is enough, like routing or short FAQ answers. Here small often lands within 2 to 3 points of the big model, for example 91 out of 100 versus 93 out of 100, while costing about one tenth and answering twice as fast. Small wins.
-
Hard tasks where the gap concentrates, like multi-step tool use. The average gap may be 8 to 10 points, but the big model fixes the same 5 to 10 hard cases small always fails. Now the upgrade is proven.
-
Mixed workloads where you split by job: small routes, big answers only the hard cases.
One rule decides whether those numbers mean anything. The judge usually needs to be stronger than the agent it grades, since grading tone or fidelity to sources is harder than lookup. Keep the judge fixed: same judge, same rubric, same cases.
Decision 2: which retrieval knobs actually help
Bigger chunks or smaller. Top 3 or top 10. Re-ranker or not. More context feels better, until evals show it made things worse.
Measure two things separately:
-
Did we retrieve the right pieces. Check with code: is the correct document in the top results. No judge needed.
-
Did the answer stay true to them. Check with a judge reading the saved chunks: does every claim come from what was retrieved.
Then change one knob at a time on the same set. Small chunks are precise but can split one answer across two chunks, while large chunks keep context but add noise.
-
A re-ranker re-sorts chunks at the price of latency. Faithfulness moving from 36 out of 50 to 43 out of 50 for 15 percent extra cost is usually a yes. One point of movement for double latency is a no.
-
Grounding and citations catch the most dangerous failure. Require a citation per claim and check with code that the chunk exists. This catches the fluent answer with no source behind it.
Change one retrieval knob at a time on the same cases. Two changes at once leave you guessing which one moved faithfulness. Per-case flips tell you what the average hides.
Decision 3: single agent or something more complex
Options run from one agent with tools to routers and handoffs between specialists. Do not start with multiple agents. Let evals prove you need the split.
Every extra agent adds handoff points where context gets lost, plus latency, cost, and debugging pain. Split only when evals show a pattern separation would fix.
Multi-step loops carry cascade risk: one small early error snowballs into a wrong outcome steps later. Catch it by evaluating trajectories, meaning tool timing and guardrail order, not just final text. If accuracy drops on cases that used to pass, the new tool added confusion. A check that runs after the answer is shown is decoration.
How to read a system comparison in words
Picture two options on the same 60 cases, 30 ticket and 30 docs cases, same judge:
-
Option A, small model with top 3 and no re-ranker: passes 47 out of 60, about 78 percent, at 8 dollars per 1,000 runs and about 2 seconds median latency. Tickets 24 out of 30, docs 23 out of 30.
-
Option B, big model with top 10 plus re-ranker: passes 50 out of 60, about 83 percent, at 16 dollars and about 5 seconds. Tickets 27 out of 30, docs 23 out of 30.
Headline says B wins by 5 points for double cost and wait. The split says more: B fixes 3 ticket cases, but flat docs hides movement as some A passes fail on B while others get fixed.
So the question is never B is 5 points better, ship it. It is: are 3 fixed ticket cases worth 2 broken docs cases plus double cost and latency. Never ship on the average alone; weight must-pass cases first and require the win to repeat.
Cost and latency belong in the scorecard
Teams grade quality first and promise to worry about cost later. Later never comes.
Treat cost and latency as first-class scores, logged per case, not just totals. Give every option quality, cost, latency, and regressions counted as flips from pass to fail. Set budgets as gates. A correct answer that took 40 tool calls is a failure with a good score.
Carry forward
-
Without evals choices run on opinion, so compare models, retrieval, and architecture on the same cases with a fixed judge and read flips plus cost and latency.
-
Small within 2 to 3 points of big usually wins on cost and speed, a re-ranker must earn its latency with clear faithfulness gains, and multi-agent must prove a pattern it fixes.
-
No ship on the average alone: weight must-pass cases first, require the win to repeat, and gate on budgets so a correct but slow or costly answer still fails.
Level 7: How to ship often without breaking trust
This level answers: how do you run all these checks in daily work without slowing the team down or burning the budget?
You now have the pieces: a test set built from real failures, cheap code checks, a judge for what code cannot see, and a way to read noisy scores. Judges are slow and cost tokens. Code checks are fast and free. So you need three habits: split checks by speed, keep costs visible, and keep a live loop after release.
Split CI by speed: fast on every commit, slow on schedule
Do not run everything on every commit. A judge call takes seconds and sometimes disagrees with itself. A code check takes milliseconds and always agrees with itself. Mix them wrong and developers stop trusting CI, then stop running it.
Run three speeds:
-
Fast code checks on every commit. Plain asserts with no model call: valid JSON, right tool with valid arguments, ticket really created, token budget kept, guardrail ran first. These live in pytest or Vitest and finish in seconds.
-
Slow judge checks per pull request and nightly. Tone, faithfulness, policy nuance, confirmation flow. A sample on each PR, the full set nightly or on the main branch.
-
Long simulations off the PR path. Multi-turn conversations and adversarial probes run nightly or weekly as early warning, never as a merge gate.
Two numbers show why the split matters. One team found 20 tickets in a row took about 162 seconds, too slow for a tight loop. After parallelizing and overlapping LLM wait time, the same run took about 18 seconds, roughly 9 times faster. Your numbers will differ. The lesson travels: when evals take minutes, developers skip them; when they take seconds, they keep them.
One rule covers all three speeds: evaluations as code. Datasets versioned in the repo, judges defined in code next to the agent, every CI run logged as its own experiment.
Important: a judge that says "good" to everything is worse than no judge. What it means for us: every metric must prove it can fail before it is allowed to gate anything. Why it matters: a passing gate everyone trusts but nobody tested is false confidence, and false confidence ships bugs.
Test your tests: the known-bad set
Judges break too. Prompts drift, models get swapped, a rubric that was strict becomes lenient. So keep a folder of conversations that are definitely bad, and demand your metrics fail them.
One team's set has 11 such fixtures: approved someone not eligible, created the ticket without asking for confirmation, quoted a ticket number it never created. Every metric must score badly on its matching fixture. If a metric says "good" there, you fix the metric, not the agent.
Two lessons from that setup:
-
The judge model must be at least as capable as the agent model. A smaller judge missed 4 to 5 of the 11 known-bad cases, so subtle errors slipped through.
-
Test each metric before it gates anything: run it against known-bad plus known-good cases, read its reasons, adjust the rubric, repeat.
One habit on top: every failure that escapes to production joins the known-bad set. This week's debugged trace becomes next month's regression test.
Control cost from the start
Evals cost tokens twice: once to run the agent, once to run the judge. Three controls keep the bill predictable:
-
Print the bill on every run. Log input tokens, output tokens, and cost per suite. When each run prints its bill, a prompt change that doubles cost is visible the same day.
-
Sample the expensive checks, run all the cheap ones. Every code check every time; a sample of judge checks. A starting point: 20 conversations per PR, 100 or more only for risky changes, 5 to 10 percent of live traffic.
-
Self-host the frequent judge. Nightly runs of hundreds of conversations add up. The most frequent judge should be the cheapest model that still fails the full known-bad set.
The live loop: ship, watch, feed back
Offline evals are snapshots. Production is weather. Close the loop in four stages:
-
Guardrails on every reply. Cheap checks before the user sees anything: toxicity, off-topic, missing citations, disallowed content. They block or rewrite; they do not grade quality.
-
Sampled judging of real threads. A slower judge scores a sample of complete conversations for satisfaction, tone, and resolution.
-
Human annotation queues. Route the uncertain cases to people: low judge confidence, guardrail blocks, user thumbs-down, anything near a safety boundary.
-
Clusters become new goldens. Group the failures, pick representatives, label the right answer, add them to the golden set.
Two stronger practices come from a team working near a regulatory line. First, per-category gating, not averaging: routing, safety, coaching quality, and summarization are scored separately, and each one can block release. Second, boundary probes block release: the safety boundary gets its own probe suite on every change, owned by a domain expert.
That team also replaced fixed scripts with simulation: a cheaper model plays the user persona against the real agent, and the transcript is judged per category. A persona that reacts to what the agent said caught jailbreaks and drove prompt hardening.
Carry forward
-
Split CI by speed so checks actually run: code on every commit, judges per PR and nightly, long simulations off the critical path.
-
Prove every judge against a known-bad set before it gates anything, and grow that set with every escaped failure.
-
Print the token bill each run, sample the expensive checks, and self-host the frequent judge once the known-bad set says you can.
-
Feed production back into goldens through guardrails, sampled judging, and annotation queues, and let per-category gates and boundary probes say no.
Tools you will meet
This is a map, not a recommendation. Every tool below can run evals, each with a stated limit. Pick by where your team already lives.
-
LangSmith — tracing, datasets, and evals in one place for LangChain and LangGraph agents. Limit: hosted pricing that grows with scale.
-
DeepEval — open-source pytest-style evals with built-in scorers and custom judges for any Python agent. Limit: judge quality depends on the model you grade with.
-
OpenAI Evals — open framework for eval suites on any model behind a predict function. Limit: examples lean toward one stack, so expect adapter work.
-
Promptfoo — CLI-first prompt and agent testing with assertions plus adversarial generation for prompts, RAG, and agents. Limit: strongest at the prompt level; multi-turn harnesses need extra wiring.
-
Inspect — open-source framework for structured evaluations on custom tasks you define in code. Limit: more harness code before the first run than hosted options.
-
Braintrust — datasets, online scoring, and experiment comparison with production logging. Limit: hosted service, so review cost and data handling first.
-
MLflow — open-source tracking of each evaluation as Dataset plus Scorer plus Predict Function. Limit: you operate the setup yourself.
-
Harbor and Terminal-Bench — sandbox benchmarks where agents run real commands in containers. Limit: heavy and slow, so nightly suites rather than every commit.
Decision guide
-
Should the agent do it at all, or should code do it? Use: a fixed rule where there is one right answer, the agent only where input varies. Then check: could a script with tests replace the agent step.
-
Code check or LLM judge for this property? Use: code for anything you can write as a rule, a judge only for what you must read to score. Then check: is there a regex, parser, or lookup that decides this first.
-
Result or steps, what do we gate on? Use: result when any path is fine, steps when order is required for safety, money, or policy. Then check: what breaks if the agent reorders two steps.
-
How many cases and repeats are enough? Use: real failures first, each job plus each must-not-do plus each must-keep-working, three runs first and more where runs disagree. Then check: does the hidden set move with the main set.
-
Is this judge ready to gate a release? Use: the known-bad set plus a blind human-vs-judge comparison on a sample. Then check: did the judge fail every known-bad case before it got veto power.
-
Did the new version really improve? Use: same cases, per-case wins and losses, repeat the run. Then check: did anything flip from pass to fail even though the average went up.
-
Can we afford these evals in six months? Use: token bills on every run, sampling for expensive judges, the cheapest judge that still fails the known-bad set. Then check: what do one PR run plus one nightly run cost in tokens and minutes.
-
What happens after release? Use: guardrails on every reply, sampled judging of real threads, annotation queues for the uncertain cases. Then check: who turns last week's failures into this week's golden cases, by name and schedule.
Takeaways
-
Normal tests fail for agents because the same input can give many valid outputs, so measure how often the agent is right over several runs instead of checking one answer once.
-
Build the test set from real failures and real user messages, covering what the agent must do, must not do, and must keep working, growing it every time a bug escapes.
-
Use cheap code checks for anything written as a rule, and save the LLM judge for what you must read to score: tone, policy nuance, clarity.
-
Use people to define what good means with examples and to audit the judge until it agrees with humans, then let automation run alone.
-
Gate on the final result when any path is fine, on the steps when order matters for safety, money, or policy, and always save the full trace.
-
Treat one score as a range, not a point: compare versions on the same cases, read per-case flips, and ship only when improvement repeats.
-
Use early evals to make system choices visible: which model is enough, which retrieval helps, which tools earn their cost.
-
Split CI by speed so checks actually run: code on every commit, judges per PR and nightly, long simulations off the critical path.
-
Prove every judge against a known-bad set before it gates anything, with a judge at least as capable as the agent, and grow the set with every escape.
-
Print the token bill on every run, sample the expensive checks, and self-host the frequent judge once the known-bad set allows a cheaper model.
-
Close the live loop after release: guardrails on every reply, sampled judging of real threads, annotation queues for uncertain cases, failures clustered into new goldens.
-
Keep per-category gates instead of one average, and give the safety boundary its own probe suite with an owner allowed to block release.
References
-
LangChain, What Is Eval-Driven Development, November 2026: https://www.langchain.com/blog/what-is-eval-driven-development
-
DeepEval, Eval Driven Development: What It Is, How to Do It Right, and Real Examples: https://deepeval.com/blog/eval-driven-development
-
Anthropic, Demystifying Evals for AI Agents, January 2026: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
-
OpenAI, Evaluation Best Practices: https://platform.openai.com/docs/guides/evaluation-best-practices
-
Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture, arXiv 2411.13768: https://arxiv.org/abs/2411.13768
-
TMLS, Eval-Driven Development for LLM Systems, June 2026: https://www.tmls.nyc/research/eval-driven-development
-
Red Hat Developer, Eval-Driven Development: Build and Evaluate Reliable AI Agents, March 2026: https://developers.redhat.com/articles/2026/03/23/eval-driven-development-build-evaluate-ai-agents
-
LangChain, Monday Service and LangSmith: Building a Code-First Evaluation Strategy from Day 1, February 2026: https://www.langchain.com/blog/customers-monday
-
LangChain, How Numan Built an Eval-First AI Health Coach with LangGraph and LangSmith, October 2026: https://www.langchain.com/blog/how-numan-built-an-eval-first-ai-health-coach-with-langgraph-and-langsmith
-
MLflow, LLM and Agent Evaluation: https://mlflow.org/docs/latest/genai/eval-monitor/



