everything you need to know about agent EVALS

i went through 46 resources on agent evals :"")

why i did this

i had an agent scoring 95% on its evals (which i wrote btw). but something was obviously wrong, and i kinda knew it before i could put my finger on the exact trace that f'ed up.

evals allow you to turn that GUT FEELING into a dashboard.

but why was the metric lying? and how should you actually evaluate an agent?

when i went looking for a quick answer, thinking there must be an A → B → C procedure i could copy paste. there wasn't any.

and now i know why

after going down a rabbit hole i found blogs (by hamel.dev the GOAT of evals), FAQs, papers, paid courses, all scattered.

i've spent hours building an intuition for evals, and YOU SHOULD TOO. reading traces is what builds intuition and im sorry but no article can do that part for you.

but maybe you don’t have those hours right now. maybe an eval is failing in prod and you need it fixed tonight.

either way, you need a map of the territory first. a BREADTH FIRST SEARCH and links to go deeper, that’s why i wrote this :)

what this is (and isn't)

this is the 80/20 of agent evals. everything you need to do your job right, nothing you don't.

i'm not covering every technique. you might want to know the rest, but you don't need it to start. if you want depth on any stage, the references at the end go deep.

you'll leave with clear next steps for your own agent.

i wanted to do something different and so i also built a small playground for this called loopbreak. it runs a fake movie-ticket agent that breaks in EVERY WAY this article talks about.

most sections end with a level you can play while you read.

use this side-by-side with this article: https://loopbreak-eval.vercel.app

(this article is also a prompt. paste it into your agent and ask “what am i doing wrong?”)

you get to judge traces on loopbreak

---

where are you right now?

you're probably in one of two places:

1. your agent is already in production, and you don't really know why it fails. → part 2

2. you're still building, and you don't know how to test it before shipping it. → part 3

first, the few ideas both paths need. i'll get into specific ROADMAPS for both situations after that. stay with me till then.

---

part 1: the concepts

evals are the ANCHOR

the first thing i want you to internalize: evals should be the ANCHOR of your workflow. every change you make to an agent should go through them.

langfuse calls this the ai engineering loop:

trace → monitor → build datasets → experiment → evaluate.

the first two steps help you understand production. the last three improve the agent during development. and you don't have to close the full loop on day one.

i think it's the right mental model to start with. but i think the real loop is bigger.

when you build an agent you're not only running evals. you're rewriting prompts, adding tools, fixing memory, changing the harness, swapping models. so i think of it as TWO LOOPS, one INSIDE the other.

the outer loop is BUILDING THE AGENT:

1. change something (prompt, tool, memory, harness, model)

2. ship it

3. watch it in prod

4. pick the next change

the inner loop is EVALS. it sits inside two of those steps:

• before you ship, the change runs against your golden dataset. if it breaks something that used to work, it doesn't ship.

• after you ship, you read traces, name the new failures, and turn them into dataset items and checks.

so step 4 never comes from a hunch. it comes from whatever the inner loop found. that's what i mean by anchor.

langchain's OG harness engineering article proved the outer loop works. they kept the model the same and only changed the harness around it. their agent jumped from 52.8 to 66.5 on terminal bench 2.0, from outside the top 30 to the top 5.

how did they know what to change? THEY READ THE TRACES.

where most people start

here is the path that i think is IDEAL for approaching evals. but people (including me before this) end up starting at step 5.

1. question requirements → read your traces

2. delete → most failures don’t need an eval

3. simplify → pass/fail, one check per failure

4. accelerate → a dataset that runs on every change

5. automate LAST → the llm judge

most teams start by picking an llm judge and a “helpfulness” (vague af) score.

that’s step 5 with steps 1 to 4 missing.

failure modes and error analysis

the ONLY way to know what to measure is to read your outputs yourself. hamel husain and shreya shankar call this error analysis and say it’s the MOST IMPORTANT ACTIVITY in evals.

also a fair warning: at some point, you WILL feel tempted to automate the labelling with an LLM. thats the worst thing you can do and will waste MORE time than just pushing through building an intuition. this is the ONE COMMON point in everything i read about evals: you cannot and should not outsource your intuition.

how to do it:

1. pull ~100 traces and open them somewhere with zero friction. if reading becomes a slog, your grading gets worse.

2. mark each trace pass or fail and write one note in your own words about what went wrong. no categories yet.

3. do the first 30 yourself. yes, by hand. i know.

4. group your notes into failure types and count them.

5. stop when the last 20 traces show no new category (langfuse’s rule of thumb).

“helpfulness” will never show up as a category. what shows up looks like “said the refund went through when the tool call failed.” those are your requirements, and every one of them actually happened.

now delete most of it.

most of that list does NOT need an eval.

• if a prompt change fixes it for good, fix it and move on.

• if the score wouldn’t change a decision, like blocking a deploy, let it go.

openai’s cookbook has the best example. a receipt-auditing system got merchant names wrong 85% of the time, and they left it alone, because it never changed whether a receipt got audited. so sort by frequency, then filter by impact.

a wrong output ONLY matters if it changes what happens next.

hard constraints are the exception. compliance and output format get a check from day one.

try it: loopbreak level 1. label reel’s conversations and find the failure types yourself.

full breakdown: part 3 of the thread.

the golden dataset

a golden dataset (GDS) is real inputs plus what a good answer looks like, written by you. start with the cases that already broke.

i used to think you can’t unit test llm outputs because they’re probabilistic. that’s only half true. code checks are unit tests for the deterministic parts. the GDS is a test suite with fuzzy assertions.

put in core features, past bugs, edge cases you already know about, and the queries your clients need to never break. anthropic says 20-50 tasks from real failures is a great start. it has to be cheap enough to run in CI on every change.

anthropic splits it in two:

• capability evals are hard on purpose. pass rates start low and you climb.

• regression evals sit near 100% and catch backsliding.

once a capability eval passes reliably, it graduates into the regression set. (so when hamel says a 70% pass rate is healthy and anthropic says ~100%, they’re both right, about different sets.)

and it’s never finished. a GDS that stops getting new failures from prod slowly stops looking like prod. skip this and you get the 95% problem from the top of this article.

try it: loopbreak level 2. add one failing conversation per failure type and watch v2 fix exactly what’s in the dataset and nothing else.

full breakdown: part 2 of the thread.

pass/fail criteria

PASS or FAIL. no 1-5 scales. NOBODY agrees on what a 3 means, and a middle option gives the model somewhere to hide. (hamel on binary vs likert)

code first. reel says “refund processed”? don’t read the transcript, check the bookings table. the transcript is what the agent SAYS. the database is what it DID.

and one check per failure. a single judge scoring everything on a 1-10 scale can’t tell you what to fix (langfuse on writing evaluators).

try it: loopbreak level 3.

full breakdown: part 4 of the thread.

llm as a judge

i wouldn't start an eval system with llm judges.

first exhaust the checks you can write in code. for everything else, where someone needs to read the output and make a judgment, an llm judge makes sense.

even then, keep it narrow. ask one question at a time, provide examples from real traces, and allow the judge to return "unknown."

one mistake i'd avoid is treating agreement with human labels as proof that the judge works.

let's say you have 100 traces. only 10 contain failures.

if the judge marks all 100 as successful, it gets 90% agreement. that's an impressive-looking number for a judge that's completely useless at detecting failures.

i'd track two things instead: True Postive Rate (TPR) and True Negative Rate (TNR)

• TPR: how often does it catch a real failure?

• TNR: how often does it correctly leave a good output alone?

a judge with 80% tpr and 90% tnr would catch 8 failures and correctly pass 81 good outputs in this example.

its overall agreement? 89%. less than the judge that never detects anything.

this is why i'd always keep a separate set of labeled traces for evaluation. hamel has a good explanation.

there's also a practical tradeoff here. i'd rather have a decent judge checking every trace than an extremely accurate one that's too expensive to run regularly.

try it yourself: loopbreak level 4

reliability: pass^k and ci

i think one of the easiest ways to overestimate an agent is to test a task once, see it pass, and move on.

agents aren't deterministic. getting the right result once doesn't tell you how often they'll get it right.

that's why i like pass^k, introduced in τ-bench.

imagine reel succeeds at a refund workflow 80% of the time.

if you run it three times and only need one success, pass@3 is about 99%. looks excellent.

require all three to pass, though, and pass^3 is just 51%, assuming independent runs.

for anything user-facing where consistency matters, that second number is far more informative.

i'd also avoid making the expected tool-call sequence part of the pass/fail criteria. there can be multiple correct ways to solve a task. grade the outcome and use the trace to find the first point of failure (hamel's article).

one more thing worth checking is whether your grader is correct.

anthropic found that grading bugs were responsible for opus 4.5 scoring only 42% on core-bench. after fixing them, the score reached 95% (source).

my approach would be to run the golden dataset in ci after every change, repeat cases, and track pass^k over time.

but ci alone isn't enough. it can only catch problems represented in your test cases. you still need production monitoring to discover new failure modes (hamel)).

try it: loopbreak level 5

i went deeper in part 5.

part 2: recipe 1, you’re already in production

you have traffic, which means you already have the best eval data there is.

1. trace everything.

2. read ~100 traces, the first 30 yourself.

3. group and count the failure types. delete what a prompt fixes.

4. write one pass/fail check per surviving failure, code first.

5. add llm judges only where you need reading, and check their TPR and TNR.

6. turn caught failures into GDS items.

7. run the GDS in CI with pass^k, and monitor live traffic.

what to watch live (langfuse on monitoring):

• code checks: valid format, PII leaks, right tool called, booking written to the db

• online judges: grounded in tool results, answered what was asked

• user signals: thumbs down, rephrasing, asking for a human, dropping off mid-booking

• ops: cost, latency, time to first token, tool calls per task

monitoring tells you which traces to read. CI decides whether you ship. keep reading 10-20 fresh traces a week.

try it: loopbreak level 1 → level 6. a prompt change ships at hour 30 and grounding failures start climbing. catch it within 3 hours without a false alarm.

---

part 3: recipe 2, you’re in early development

no users yet, so you build the dataset first and carry it into prod later.

1. build a small GDS from expected scenarios plus anything that’s broken so far. 20-50 tasks is plenty.

2. write pass/fail checks for anything that must never break.

3. run the agent and read every output yourself.

4. add a judge only where you need one, and validate it.

5. run it in CI with pass^k.

6. turn on tracing from day one.

how early should evals come? anthropic says write them before the feature exists. hamel says “generally no.” i land in the middle: hard constraints get a check on day one, everything else waits until it fails.

no traffic? don’t ask an llm for “50 test questions.” pick dimensions (persona, intent, edge case, mood), combine them, then turn each combo into a real-sounding message (hamel’s method). and for multi-turn agents, let an llm play the customer with a hidden goal and grade the final database. that’s how reel works.

try it: loopbreak levels 2 → 3 → 5.

---

WRAP-UP: back to the 95%

so why did an agent scoring 95% still fail users? i’d look for four things:

1. the GDS went stale. nobody fed it new failures from prod.

2. the evals graded what the agent said. “refund processed” passes even when the refund never happened.

3. the judge had high agreement and low TPR. it passed almost everything.

4. the score was pass@1. users hit the same task again and again.

eugene yan put it best: “automated evaluators amplify our existing annotation and feedback processes.”

rough next steps for you:

1. pull 100 traces

2. read 30 yourself

3. group and count the failure types

4. delete what a prompt fixes

5. build one check for the worst one left

6. measure it

bonus: jev, a judge that can’t write a sentence

typesafe’s jev is a “decision model.” you give it a trace and a yes/no or multiple-choice question, and it gives back a probability. no reasoning, no text. it costs a tiny fraction of an llm judge, so running it on EVERY trace is realistic (langfuse already supports it).

the result that matches this whole article: on 2,000 phishing emails, jev scored 62.6% on one big question. split into five narrow questions, it hit 95%. one check per failure, again.

what's next

next i'm trying decision-model judges on voice agents. they judge the transcript and NOT the audio, so i want to find out what they miss. follow along at @luhsnaa.

the full evals thread: part 1 · part 2 · part 3 · part 4 · part 5 · part 6

references

if you only read five:

1. hamel husain & shreya shankar: AI evals FAQ

2. langfuse: ai engineering loop

3. anthropic: demystifying evals for ai agents

4. langfuse: writing good evaluators

5. eugene yan: product evals in three simple steps

all 46 resources

langfuse (9)

• AI Engineering Loop

• Evaluation

• Choosing what to evaluate

• Writing good evaluators

• Monitoring

• Designing great datasets

• Error analysis cookbook

• AI agent evaluation

• Using TypeSafe's Jev for evals

hamel husain and shreya shankar (16)

• AI Evals FAQ

• How do I evaluate agentic workflows?

• Evals in CI vs production monitoring

• How do I know if I can trust my automated eval?

• Binary vs Likert

• Ready-to-use metrics

• An evaluator for every failure mode?

• Evaluating RAG

• Same model as judge

• Combining evals into one metric

• Guardrails vs evaluators

• Should I practice eval-driven development?

• Generating synthetic data

• Your AI Product Needs Evals

• Using LLM-as-a-Judge

• Can I use Jev for evals?

eugene yan (3)

• Evaluating the Effectiveness of LLM-Evaluators

• An LLM-as-Judge Won't Save the Product

• Product Evals in Three Simple Steps

labs and platforms (6)

• Anthropic: Demystifying evals for AI agents

• OpenAI cookbook: Eval-driven system design (receipts)

• LangSmith: Evaluation concepts

• LangSmith: Evaluation approaches

• Hugging Face agents course, Bonus Unit 2

• Hugging Face: Agent observability and evaluation

harness engineering (2)

• LangChain: Improving Deep Agents with harness engineering

• Birgitta Böckeler: Harness engineering for coding agent users

jev and decision models (4)

• Arize: TypeSafe's Jev: Can decision models replace LLM judges?

• Firecrawl: What Is Jev? Inside TypeSafe's Decision-Only AI Model

• inovex: TypeSafe AI Jev Review

• BERI: TypeSafe's Jev Scores 62.6% Asked Once and 95% Split Five Ways

papers and benchmarks (6)

• Shankar et al., Who Validates the Validators?

• Yao et al., τ-bench

• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

• Shreya Shankar, In Defense of AI Evals, for Everyone

• τ-bench leaderboard

• Sierra, τ^τ-bench (hyper-τ-bench)