How to write and test agent skills with evals

For engineers who run coding agents and decide which instruction files to install or to write. After reading, you can take any skill, pull out the ideas it rests on, and measure in two numbers whether the agent follows it and whether following it helps, before it touches your prompt. The same two questions serve the people who approve a marketplace or an internal skill library, because they are the questions a budget holder should ask about any prompt that runs on every task. The figures are published studies listed at the end, plus one public skill file that is read in full in Level 0, so nothing here depends on a setup you cannot see.

The quick grasp (read this if you read nothing else)

A skill is a text file of instructions that an agent reads before or during a task. Nothing in the model changes when a skill is installed. The model gets more text in its context, and it behaves differently because of that text, in the same way it behaves differently after any other paragraph you paste into a prompt.

Why skills exist: people want an agent to do a type of task in a particular way, every time, without retyping the instructions.

The problem: more than a million of these files are now shared in public marketplaces, and the shared format asks for a name and a description, not a test. The typical skill is written from what felt right, used a few times, and published. No tests, no evals, no metrics, so whether it helps, hurts or only costs tokens is an opinion. And a skill that was tested was tested in one person's setup. Context differs from person to person: the rules files, the other skills in the listing, the tools, the repositories, the way work is handed to the agent. A test in one environment is not a test in yours.

This is how a skill should be created:

1. Objective in one sentence. What must be better when the skill is used, stated as something that can be measured. If it cannot be measured it cannot be studied, and the skill is a subjective piece of text.

2. Ideas as points. A short list of core points, each saying what the agent must do and carrying a hypothesis for why that helps. Secondary points the same way. If no hypothesis can be written, the point is folklore.

3. Points into an instruction. In this step the points are turned into the instruction text the agent will read. Every sentence in the file comes from a point; the points are what gets shared and reviewed, the text is how they are delivered.

4. Metrics. Adherence, one check per point over the log of what the agent did. Effect, on real tasks with an answer the agent cannot see: a hidden check, plus a quality label for each thing the agent produced (for tests: critical, useful, nitpick, useless or harmful) from a judge checked against your own hand labels, plus honesty, cost and stability. Quality of the output, not the amount of it.

5. Evals. Your tasks, your context, your model. Arms: no skill, the skill, a block of unrelated text of the same length. Several runs each.

6. Measure, decide, ship. Low adherence means rewrite the text; high adherence with a flat effect means drop the idea. Ship the text with its points, evals and results, and re-run after each model update.

This is how an external skill should be validated before it goes anywhere near your prompt:

• Decompose the instruction into ideas. Read the skill text and extract its points with a hypothesis each. If none can be found, stop: there is nothing to test, and installing it buys obedience to nothing.

• Write the metrics. One adherence check per point; the effect measures above.

• Create the evals. The same arms as above, on your tasks, in your context, on your model, because nobody else's test was run there.

• Measure and decide. Keep, cut or drop, per model, and repeat when the model or your context changes.

Important: a skill fails in two different ways, and the outcome alone cannot tell you which. Either the agent does not do what the text says (bad text), or it does and the result is no better (bad idea). What it means for us: every skill needs two scores, adherence and effect. A low adherence score sends you back to the wording; a high adherence score with a flat effect sends you back to the idea. Why it matters: teams that only track outcomes rewrite good ideas because the wording was weak, and keep obeying bad ideas because the wording was strong.

The rest of this article zooms in, one level at a time. Each level answers one question about the level above it.

• Level 0 · The method once, end to end, on one point of one public skill

Part 1: the problems

• Level 1 · What does a skill cost before it does anything?

• Level 2 · Does anyone test shared skills?

• Level 3 · Does a skill carry over from one model, or one setup, to another?

• Level 4 · Why "it worked for me" is not evidence

Part 2: the method

• Level 5 · What is the skill for?

• Level 6 · What are its ideas?

• Level 7 · How do we know the agent did it?

• Level 8 · How do we know doing it helped?

• Level 9 · How do we run the evals?

• Level 10 · How do we decide, ship, and keep it alive?

Part 3: the use case

• Level 11 · The example skill taken through the whole method

• Then · The tools and approaches, a decision guide, and the references

Level 0: the method once, end to end, on one point of one public skill

This level answers: what does testing a skill look like when it runs?

The running example, introduced once. The skill used throughout is the test-driven-development skill from a public repository called superpowers, one of the most starred skill collections for coding agents. Everything the reader needs is in the public files, and the numbers below can be re-counted from them:

• The repository holds 15 skills and a session hook. The hook injects a 492-word "using-superpowers" file at the start of every session. That file tells the agent that if there is "even a 1% chance a skill might apply" it "ABSOLUTELY MUST invoke the skill".

• The test-driven-development skill is a single file of about 1,475 words, with a companion file of about 1,310 words on writing good tests that the skill asks the agent to read whenever it writes a test.

• Its one-line description says it applies to "any feature or bugfix, before writing implementation code". Its first rule, called the Iron Law, reads: "NO PRODUCTION CODE WITHOUT A FAILING TEST FIRST".

• The code examples in it are TypeScript run with npm. The tasks used to test it in this article are Python repositories, so the examples never match the language the agent is working in.

The method on one point. Take the single most distinctive instruction in the file: write the test, run it, and watch it fail before writing any code. Here is the whole method applied to that one point.

• Objective. When the agent changes code, the change must pass tests it never saw, and the tests the agent wrote must fail before the change and pass after it.

• Point and hypothesis. Point: run the new test and watch it fail before writing code. Hypothesis: a test that was never seen failing may pass for the wrong reason, so watching it fail removes tests that prove nothing.

• Adherence check. From the log of commands: a test command runs after the test file is written and before the first source file is written, it exits non-zero, and the failure names the new test rather than a syntax error. Yes or no, no judgement needed.

• Effect check. After the run, apply the task's hidden tests. Then run every test the agent wrote against the code before the change and against a few planted faults, and label each one critical, useful, nitpick, useless or harmful. A test that never fails is decoration.

• Comparison. The same tasks run three times each with no skill, with the skill, and with an unrelated text of the same length. Each run reports hidden-test pass rate, share of critical tests, tokens, turns and minutes.

• Verdict rule. If adherence is high and the share of critical tests rises against both comparisons, the point stays, with its extra tokens and turns recorded next to the gain. If adherence is low, the wording is at fault and the point is not yet tested. If adherence is high and the share of critical tests does not move, the point is an idea that does not work for this model, however good it sounds.

Every later level takes one piece of this apart.

Carry forward from Level 0:

• A skill is a hypothesis about agent behaviour, written as text.

• It needs two scores: did the agent do it, did doing it help.

• Both scores need a comparison, and the comparison needs a control for length.

Level 1: what does a skill cost before it does anything?

This level answers: what does the agent pay for a skill that is merely installed?

A skill costs tokens in three places. Its name and description sit in the prompt of every turn so that the model can decide to open it. Its body is loaded when it is opened and stays in the conversation afterwards. And some collections add a hook that injects a file at every session start whether or not any skill is relevant.

The listing is the part people forget. The Claude Code documentation says it plainly: "every line is a recurring token cost" (Anthropic, Claude Code skills documentation). One developer machine, checked with the built-in skill-doctor report:

• 89 skills in the listing, from the user's own folder, from synced cloud skills and from installed plugins.

• 81 of the 89 had never been invoked.

• The 8 that had been used were last used 37 days earlier at best.

The listing exists so that the model can select, and the evidence says it mostly does not:

• Vercel reports that its default skill was never invoked in 56% of eval cases, and that the score with the skill installed was 53%, the same as with no documentation at all; an explicit "invoke this skill" line raised it to 79% and an always-on documentation index reached 100% (Vercel, 2026, a vendor claim with model and case count undisclosed). The gain came from forcing the load, not from the skill's content.

• An independent re-test on 22 prompts with Claude Sonnet 4.5 found 50% to 55% activation without a forcing hook and 100% with one (Spence, 2026). Same picture: the native selection step fires about half the time.

• A benchmark of 5,400 cases over eight models measured how often each model loaded any skill at all: between 11.1% and 72.1% depending on the model, and about as often on tasks the model could solve alone as on tasks it could not (Skill Retrieval Augmentation, arXiv 2604.24594). The authors call the models "not need-aware": the decision to load does not track whether loading would help.

• As the pool grows the choice gets worse: the share of loaded skills that were the right ones fell from 29.6% with 5 skills to 3.3% with 100 (Demystifying Agent Skills, arXiv 2608.14036). A long listing does not just cost tokens, it lowers the chance of a correct pick.

The hook is the other hidden charge. In the example, 492 words land in every session before the user has typed anything. That is a loading policy, not a skill, and it pays for itself only if the skills it forces are good.

Important: the listing exists so the model can choose a skill, and across vendor evals, independent re-tests and an eight-model benchmark the model chooses correctly about half the time or less. What it means for us: either name the skill in the task, route it with a hook or a naming rule, or accept that half of the runs are the no-skill baseline with extra tokens. Why it matters: every measurement of a skill's effect that does not record whether the body was loaded is a measurement of a mixture.

Carry forward from Level 1:

• Listing, body and hook are three separate costs, paid on different schedules.

• The native selection step fires about half the time and gets worse as the pool grows.

• A test of a skill must say how the skill was loaded, or its numbers mix two conditions.

Level 2: does anyone test shared skills?

This level answers: when a skill is published, what has been checked about it?

The public specification for skills defines a folder layout, a name, a description and how the body is loaded (agentskills.io specification). It does not define a test, an eval or a quality bar. I found no published count of how many shared skills ship with any eval at all. That absence is itself the finding.

What happens when researchers test shared skills on real tasks:

• 31 public skills from well-known repositories were injected into every prompt on 50 web projects across four models. All four models lost pass rate, with Claude Sonnet 4 down 4.2 points at two attempts, and tokens rose between 72% and 394% depending on the model (Signal or Noise, arXiv 2608.23067).

• Only 1% of skill-and-project pairs gained on all four models in that study. A skill that helps everywhere is the exception.

• A separate benchmark of 49 public software-engineering skills found 39 of the 49 gave zero gain, an average gain of 1.2 points, and token overhead up to 451% (SWE-Skills-Bench, arXiv 2603.15401). Three skills hurt by up to 10 points, and the authors trace the harm to guidance written for an older version of the library.

• A study of 307 cases where a skill changed the outcome found 182 were cost regressions with no gain (Agent Skills Can Be Harmful, arXiv 2608.11888). These are selected failures, not a base rate, but the common shape is the same result at a higher bill.

Security is the extreme form of untested. A skill is a file that an agent reads and often a script it runs, so sharing one is sharing code:

• Of 31,132 marketplace skills scanned, 26.1% had at least one vulnerability pattern, and skills that bundle scripts were 2.12 times more likely to be affected (Agent Skills in the Wild, arXiv 2601.10338). One in four is not a tail.

• Among 98,380 scanned skills, 157 were confirmed malicious, more than half from a single actor (Do Not Mention This to the User, arXiv 2602.06547, USENIX Security 2026). The gate was the researchers, not the platform.

• Five skill-scanning products were bypassed in a published test, three of the four bypasses in under an hour (Trail of Bits via Cloud Security Alliance, 2026). A scan is not a review.

Important: the skill format defines how a file is loaded, not whether it works, and the measured record of shared skills is zero gain for most, a token bill for all, and a security rate of one in four. What it means for us: treat a downloaded skill the way you treat a pull request from a stranger. Read it, run its evals if it has any, write them if it does not, and keep it out of the trusted folder until they pass on your model. Why it matters: installing is the step that puts the text into every task, and the format attaches no check to it.

Carry forward from Level 2:

• Published means formatted, not tested.

• The measured default for a shared skill is no gain and more tokens, with stale guidance as the main source of actual harm.

• Sharing a skill is sharing code, and the scans show the vulnerability rate of unreviewed code.

Level 3: does a skill carry over from one model, or one setup, to another?

This level answers: if a skill was tested on one model in one environment, what do we know about it on another?

A skill is a prompt, and prompts do not transfer. The same text produces different behaviour in different models, and the difference is not a small correction:

• The same skill on the same project swung 33 points up on Claude Sonnet 4 and 22 points down on DeepSeek and Qwen, and across all pairs the correlation of gains between models was about zero (Signal or Noise, arXiv 2608.23067). Knowing a skill helped one model tells you nothing about the next.

• With a curated set of skills, Claude Haiku 4.5 scored 27.7% against 22.0% for Claude Opus 4.5 without them (SkillsBench, arXiv 2602.12670). A good skill can lift a cheap model over an expensive one, which is exactly why the gain has to be measured per model rather than assumed.

The model is only the first axis. A skill never runs alone. It runs next to the user's rules file, the other skills in the listing, the tools the agent has, the repository it is in, and the way that particular person hands work to the agent. Everyone's context is unique, and everyone's way of collecting and feeding context is unique, so the same skill meets a different prompt in every environment:

• In the injected-skill study, only 1% of skill-and-project pairs gained on all four models (Signal or Noise, arXiv 2608.23067). Whether a skill helped depended on the pair of project and model, not on the skill alone.

• In Vercel's report the skill scored 53%, the same as no documentation, while an always-on documentation index scored 100% (Vercel, 2026, a vendor claim). What is already in the context, and how it got there, decided more than the skill did.

A marketplace skill carries no model label and no environment label; the specification has no field for "tested on". A skill that someone tested was tested in that person's setup, and that setup is not yours.

Important: a skill is accepted per model and per environment, not in general. The same file can be a gain on one model and a loss on another, and a gain in one setup and a loss in yours. What it means for us: an eval of a skill is a cell in a matrix. Run it for the model you use, in your context, on your tasks, and run it again when the model or the context changes. Why it matters: "we tested it" without "on which model, in which setup" does not describe the skill you will run.

Carry forward from Level 3:

• Same text, different model or different setup, different sign.

• Adherence and effect depend on the model and on everything else in the context.

• Someone else's test, however honest, was run in someone else's environment.

Level 4: why "it worked for me" is not evidence

This level answers: what is wrong with the way skills are judged today?

The acceptance test for a vibe-written skill is its author's feeling after a few sessions. That is one run, on one model, with no baseline, no cost count and no repeat. Each of those gaps hides a specific error:

• A careful study that measured the gain of a skill library at 2.8 points reported a confidence interval that crossed zero (Demystifying Agent Skills, arXiv 2608.14036). A single run cannot tell a 3-point gain from noise; a feeling cannot tell a 3-point gain from a 3-point loss.

• Of the 307 skill-induced harm cases in the harm study, 62.6% were cost regressions rather than failures (Agent Skills Can Be Harmful, arXiv 2608.11888). A skill that makes the agent slower and more expensive while producing the same answer feels fine in the chair and shows up only on the bill.

• Where skills demonstrably work, someone measured them. Anthropic reports an internal analytics agent going from 21% to above 95% accuracy on its eval set after skill files were written and maintained against a few dozen cases per topic (Anthropic, 2026, a vendor claim). The gain came with an eval.

• The mechanism of a good skill is procedural: in the study that looked inside, 65.7% of the benefit came from anchoring the agent to a procedure, and 4.5% from new knowledge (Demystifying Agent Skills, arXiv 2608.14036). A skill works by making the agent follow steps, which is why its steps are what has to be checked.

This is where the two failure modes from the quick grasp come from: the agent obeys the text and nobody checks whether obeying helped, or it ignores the text and nobody notices. Outcome alone cannot tell the two apart.

There is a deeper problem with how skills are shared. People share text when they should share ideas. A skill's ideas are a handful of statements of the form "do X, because it makes Y better". A reader can judge a handful of those quickly. Fifteen hundred words of instructions cannot be judged by reading them, which is why they are judged by feel. Some shared skills, read closely, contain no idea at all: they are tone, urgency and formatting rules. Installing one of those buys adherence to nothing, which costs tokens at best and distorts behaviour at worst.

Important: a skill is a hypothesis, usually untested, and in the studies that did test shared skills most of them gave no gain. What it means for us: share and review skills as their ideas, a few points with a reason each, and keep the text as the implementation of those points. Accept a skill only with its evals. Why it matters: text without ideas cannot be evaluated, only obeyed, and obedience without a reason has a token cost and no measured benefit.

Carry forward from Level 4:

• One run, no baseline, no cost count: the feeling cannot see a 3-point change, and it never sees the bill.

• Skills work by anchoring a procedure, so the procedure is what to test.

• Share the ideas; the text is one way to write them down.

Level 5: what is the skill for?

This level answers: what is the first sentence to write, before any instruction?

Every skill starts from one sentence: what must be better when the skill is used. The sentence names an outcome that can be measured. If it cannot be measured it cannot be studied, and the skill stays what most skills are today: a subjective piece of text. Everything written later is checked against it, and a rule that does not move it is style, not a point.

The sentence does not set a price. The evidence from Level 2 is that skills routinely multiply the token bill, so the price is measured in the evals and reported next to the gain, and whether a given trade of accuracy for tokens is worth it is decided then, for that task, with both numbers in view. Nobody can set that trade in advance, and pretending to only hides the bill.

For the example skill, the objective its text implies is:

When the agent changes code, the change is correct (tests it never saw pass, and nothing that passed before now fails) and comes with at least one critical test per behaviour (a test that fails before the change and checks what the task asked for),

Notice what this sentence does. It turns "do test-driven development" into two things that can be measured: hidden tests passing, and critical tests present. Anything the skill asks for that moves neither is not part of the objective.

Carry forward from Level 5:

• One sentence, one measurable outcome; unmeasurable means unstudied.

• Cost is not in the sentence; it is measured in the evals and reported next to the gain.

• Every later rule is checked against this sentence.

Level 6: what are its ideas?

This level answers: how do we turn a skill into a short list of things we can test?

The ideas of a skill are its points. A core point is a statement of what the agent must do, with a hypothesis for why that helps, written as a testable claim: "doing X makes outcome Y better because Z". Keep the list short enough that a reviewer can read it in a minute. If the hypothesis cannot be written, the point is folklore, and the honest thing is to label it as such. Secondary points are supporting rules in the same form, reported separately and first to go when the text has to shrink.

Points come from three sources, listed by trust:

• Failures seen in the baseline. Run the agent without the skill on real tasks, read what went wrong, write a point for each failure that repeats. The study of playbooks built from an agent's own failures reports a 12.3% gain on an agent benchmark from exactly this loop (Agentic Context Engineering, arXiv 2510.04618). Points from failures are hypotheses with evidence attached.

• Measured evidence, from published studies or from earlier evals you ran.

• Doctrine, how experts say the task should be done. Doctrine points get written down with their hypothesis and a label saying they are untested. The running example is doctrine from top to bottom.

For the example skill, reading the text yields seven core points. Each is quoted or paraphrased from the file, and the hypothesis is the claim the file makes or implies:

• C1. Write the test before any production code. Hypothesis: stating the expected behaviour first makes the test catch the real bug instead of the agent's guess at it.

• C2. Run the test and watch it fail, for the right reason. Hypothesis: a test never seen failing may be passing for the wrong reason, so this step removes tests that prove nothing.

• C3. Write the minimal code to pass, run, and watch it pass. Hypothesis: less speculative code means fewer regressions.

• C4. When a test fails, fix the code, never weaken the test. Hypothesis: stops the agent from turning a red test green by deleting the assertion.

• C5. Run the full suite before claiming done, and name any failure. Hypothesis: catches regressions and forces an honest final report.

• C6. A bug fix starts with a test that reproduces the bug. Hypothesis: the fix is then proven on the real bug, not on a symptom.

• C7. Small cycles, one behaviour each. Hypothesis: failures are isolated and fewer turns are wasted.

The secondary points are the test-quality rules: one behaviour per test, no mocks of the unit under test, every new function gets a test, cover edge cases, refactor only after green, keep test output clean.

The instruction text is written from these points and nothing else: every sentence in the file comes from a point, and examples use the language and tools the agent will meet. The text is how the points are delivered; the points are what is shared and reviewed.

Two more things in the file are neither core nor secondary. The first is a block of arguments against the agent's own excuses, a table of rationalisations and a list of red flags. It is not an instruction about the task but a device meant to raise adherence to C1, with the hypothesis "pre-empting excuses makes the agent follow the Iron Law more often", and it can only be tested by the adherence family. The second is the session hook, a loading policy, tested by comparing the skill with and without it.

Important: the points are the skill. They are what gets shared, reviewed and evaluated; the text is one way of writing them down. What it means for us: before installing a skill, extract its points. If you can write seven sentences with a hypothesis each, you have something to test. If you cannot find one, stop: there is nothing in the file to evaluate. Why it matters: a reviewer can disagree with a point. There is nothing in fifteen hundred words of instructions to disagree with until the points are pulled out.

Carry forward from Level 6:

• A short list of core points, each with a testable hypothesis; secondary points reported separately.

• Points from baseline failures outrank points from doctrine.

• Persuasion text and loading hooks are points of a different kind and get their own tests.

Level 7: how do we know the agent did it?

This level answers: how is adherence measured without a human reading every transcript?

Adherence is the first metric family: did the agent do what each point says. It is graded from the harness log, meaning the record of what the agent did during a run: every file it wrote, every command it ran with its exit code and output, and the final message it sent. The headless mode of a coding agent prints it as a stream of events, and a hook on the skill tool records whether and when a body was loaded.

One grader per point, and the grader is a rule in code, not a judgement. For the example:

• C1, test first. Met when the first write to a test file comes before the first write to a source file in the task.

• C2, watch it fail. Met when a test command runs after the test write and before the first source write, exits non-zero, and its output names the new test rather than an import or syntax error.

• C3, watch it pass. Met when a test command runs after the source write and exits zero.

• C4, never weaken the test. Met when the agent's final test file still fails against the code as it was before the change, and no test was deleted after it ran red.

• C5, full suite before done. Met when a whole-suite command runs after the last source write and the final message mentions any failure that run showed.

• C6, reproduce the bug first. On bug-fix tasks, met when a test that fails on the original code exists before the first source write.

• C7, small cycles. Counted as the number of write-then-run alternations and the number of source edits per test run.

The adherence score of a run is the share of applicable core points met. Secondary points get their own share, reported but never in the headline. The grader for the session hook is simpler still: was the skill body loaded, and on which turn.

Three limits, so the metric is not misread:

• Adherence measures obedience, not benefit. A perfect score says the text worked as text, nothing more.

• Code graders see actions, not intent. An agent that runs the test and ignores the output satisfies C2 as written; the C4 grader, which checks that the final tests still fail on the old code, is what catches it. Points are graded as a set for this reason.

• Activation is part of adherence. A skill that is not loaded scores zero on every point, and that zero must stay visible rather than be averaged away.

Carry forward from Level 7:

• One code grader per point, over the log of actions.

• The score is a share of points met, plus a loaded-or-not flag.

• High adherence proves the text; it does not prove the idea.

Level 8: how do we know doing it helped?

This level answers: how is the effect measured so that it rewards good tests rather than many tests, and how is it tied back to the points?

Effect is the second family: did the result get better. The easy version counts things: tests written, tests passed, lines covered. For a test-first skill that is wrong, and there is a study that shows why. An analysis of agent trajectories on SWE-bench Verified found that tasks the agents solved and tasks they failed showed the same test-writing frequency, that agents checked values with print statements more often than with assertions, and that prompting four models to write more or fewer tests did not change their outcomes (Rethinking the Value of Agent-Generated Tests, arXiv 2602.07900). Counting tests measures effort, not benefit. Coverage is no better: a replication on model-written suites found coverage and mutation scores unreliable once the code under test is itself buggy (arXiv 2607.22880), which is the state of the code during a bug fix. And across more than 6,000 faulty programs written by models, generated tests often executed the faulty line and still passed, because the assertion did not check the thing that was wrong (arXiv 2609.09315).

So the effect measure is built the other way round. First decide what a good test looks like for your team, with no skill in the room. Then grade every test the agent wrote against that definition, one test at a time. This is a validation framework for test-first work: it defines the target, and a skill is one attempt to hit it, as is "no skill".

The framework has three parts.

• Ground truth. The task's hidden tests, which the agent never saw, plus the existing suite. They gate everything: a run whose hidden tests fail has no effect to discuss. They are not enough on their own, because a fixed test set can be passed while parts of the intended behaviour stay unimplemented (GTDD, arXiv 2610.02952).

• One label per test. Every test the agent wrote gets a label from a fixed set. Critical: it fails on the code before the change and checks the behaviour the task is about. Useful: it checks a boundary or error path of that behaviour that a reviewer would ask for. Nitpick: it checks something true but unimportant, or repeats another test. Useless: it cannot fail, because it has no assertion, asserts on a mock, prints a value, or restates the implementation. Harmful: it pins down wrong behaviour, or was edited until it passed.

• Evidence before judgement. The labels are anchored to runs, not to reading. Each test is executed against the code before the change and against a few small planted faults in the changed lines. A test that passes on all of them cannot be critical or useful, whatever its name says, and goes straight to useless. Only tests that fail on something reach the judge, which separates critical from useful from nitpick by reading the test next to the task text.

The judge is a model with a rubric, validated like any other judge: someone on the team labels a sample of tests by hand, agreement is measured, and the rubric is rewritten until the agreement is good enough to trust. Until then the labels are an opinion.

What is reported per run. The share of tests under each label, and one headline: how many of the task's behaviours have at least one critical test. Quality over quantity means a run with two critical tests beats a run with fifteen tests and one critical. Next to it: honesty (did the agent claim done, and do the hidden tests agree), cost (turns, tokens, minutes and money, against the no-skill run) and stability (several runs per task, pass rate on the first attempt and within three, spread of cost).

Why each part is there:

• Without a hidden check, the agent grades its own homework.

• Without a label per test, a skill wins by producing more tests, and the trajectory study above says more tests do not resolve more tasks.

• Without the run against planted faults, the judge grades a test by how it reads. One practitioner's parser had 100% line coverage and a 61% mutation score (Mutation Testing for Agent-Written Code, awesome-testing.com, August 2026): a test that reads well and checks little is not found by reading.

• Without cost, 62.6% of the harm in the harm study is invisible, because it was cost rather than failure (Agent Skills Can Be Harmful, arXiv 2608.11888).

• Without repeats, a 3-point change is noise: the 2.8-point library gain with an interval crossing zero (Demystifying Agent Skills, arXiv 2608.14036) is what one-run evaluation looks like once someone computes the interval.

The framework also pays back into Level 6. The baseline's label distribution is the list of points to write: if the no-skill runs produce useless tests because they print instead of assert, the point is "assert, do not print", and it came from a measured failure rather than from doctrine.

Attribution connects the two families. Across tasks within the skill arm, correlate each point's adherence with the critical-test count. A point that is obeyed and moves nothing is a candidate for removal; a point rarely obeyed but tied to critical tests when it is obeyed is the one to rewrite first.

Ablation confirms it. Remove one point's text, keep everything else, re-run both families. If adherence to that point drops and the effect does not, the point goes. If the effect drops, the point stays, however few words it takes.

This is where the two failure modes become a decision rule:

• Adherence low: the text is at fault. Rewrite the wording, not the idea, and re-run.

• Adherence high, effect flat against baseline: the idea is at fault for this model. Drop the point or mark it untested.

• Adherence high, effect up: keep if the cost is one you accept with the gain in front of you, and record both numbers with the skill; otherwise cut secondary points and measure again.

Important: the definition of a good test is written before any skill is looked at, and it never mentions one. The skill is graded against your definition, not against its own promises, and so is the baseline that has no skill at all. What it means for us: build the rubric, the planted faults and the judge check first. Then every arm, including "no skill", gets the same per-test labels, and the question "did the skill help" becomes "did the share of critical tests go up at a cost we accept". Why it matters: without a definition of good, any skill that makes the agent write more tests looks better, and the published trajectory study says that is the one thing that does not help.

Carry forward from Level 8:

• Effect for a test-first skill is a label per test, anchored to runs against the old code and planted faults, then read by a validated judge.

• The headline is behaviours with a critical test, next to honesty, cost and stability.

• The framework is defined without the skill; attribution and ablation tie it back to the points.

Level 9: how do we run the evals?

This level answers: what does the experiment itself look like, and how big does it have to be?

The eval set is real tasks with hidden ground truth, a few dozen of them. Anthropic reports working with a few dozen eval cases per topic (Anthropic, 2026, a vendor claim). With fewer tasks and no repeats, the interval covers any effect of the size the published studies report, as the 2.8-point gain with an interval crossing zero in Level 4 shows.

For a coding skill the natural source is a benchmark of real repository tasks collected after the model's training cutoff, each with a problem statement, hidden tests that fail before the fix and pass after it, and the repository's existing tests (SWE-bench-Live, arXiv 2505.23419; SWE-rebench, arXiv 2505.20411). The agent never sees the hidden tests.

The arms, every one of them invoked explicitly so that the activation problem from Level 1 does not contaminate the effect measurement:

• Baseline. No skill. Its failures are where the points for a skill you write come from.

• The skill, as the reader would install it.

• The skill body alone, without its session hook, if the collection has one. The difference between this arm and the previous one is the price of the hook.

• An alternative procedure that reaches the same objective another way. For a test-first skill, the obvious alternative is code first, tests after, then verify the tests fail on the old code. If this arm matches the skill, the idea that matters is "verify your tests", not "write them first".

• A length control. A block of sensible but unrelated text, the same token count as the skill. Extra text alone changes behaviour: the equally long irrelevant skill cost 3.3 and 3.5 points on two models in the injected-skill study (Signal or Noise, arXiv 2608.23067). Without this arm, a loss may be the length and a gain may be "more tokens", and neither says anything about the ideas.

Run rules: a fresh session per run, at least two runs per task per arm so there is a spread to report and three as the usual choice, the harness log captured for every run, and everything per model, with the row re-run when the model version changes.

The budget is arms times tasks times runs: twenty tasks, five arms and three runs is 300 runs. Set a money ceiling before the first run, because the cost of a skill eval is itself a number worth knowing about the skill.

Important: the length control is the arm that is easiest to skip, and it is the one that separates "these ideas helped" from "any extra text would have done this". What it means for us: add the control, even when it feels silly. A skill that beats baseline and loses to its own length control has no ideas in it that the model can use. Why it matters: the published losses from skills are largely length losses, so a skill that is not compared against its own length is being graded on a curve.

Carry forward from Level 9:

• A few dozen real tasks, hidden checks, explicit invocation.

• Baseline, skill, alternative procedure, length control; three runs; per model.

• Budget is arms times tasks times runs, with a ceiling set first.

Level 10: how do we decide, ship, and keep it alive?

This level answers: what happens after the numbers come in, and then every month after that?

The decision rule is the one from Level 8, applied against both comparisons. Accept a skill only if, against the baseline and against the length control, the effect improves, the extra cost is one you accept with the gain in front of you, and adherence explains the effect. Otherwise rewrite the text when adherence is low, or drop points when adherence is high and the effect is flat, and run again. Every version is kept with its numbers.

A skill ships as a bundle of four things: the text, the points with their hypotheses, the eval set, and the results table with the model and version it was run on. Anything less is a file with a claim attached.

The bundle has to be re-run, because adherence drifts and nobody announces it:

• Anthropic reports accuracy drifting from about 95% to about 65% within a month when skill files were left alone while the underlying data changed (Anthropic, 2026, a vendor claim). The text did not change; the world under it did.

• The three skills that hurt by up to 10 points in the 49-skill benchmark hurt because their guidance matched an older library version (SWE-Skills-Bench, arXiv 2603.15401). Stale is a form of wrong.

• Agent-written skill systems that work have a test gate on every rewrite: one reports a score rising from 52.3% to 66.0% with the gate (Memento-Skills, arXiv 2603.18743), another loses 6.2 to 11.5 points without it (Meta-Context Engineering, arXiv 2607.16621).

So the schedule is: re-run after each model update, after each major change to the tools or libraries the skill talks about, and on a calendar otherwise.

For an existing skill the same steps apply, entered from the text instead of from the objective. Write the objective the skill implies. Extract its points with a hypothesis each, and label the persuasion text and any loading hook as separate points. If no point can be extracted, stop: there is nothing to evaluate, and the skill should not be installed. Otherwise build the evals, run the arms, and give a verdict per model: keep, cut, or drop.

When you publish, publish the bundle.

Important: a skill is never finished; it is accepted for a model and a date, and the acceptance expires. What it means for us: the eval set is the asset. The text is replaceable, the points are stable, and the numbers have a shelf life. Why it matters: a one-month drift from 95% to 65% is the published cost of treating a skill as done.

Carry forward from Level 10:

• Accept on both comparisons, with the cost next to the gain and adherence explaining effect.

• Ship text, points, evals and results together.

• Re-run on every model or dependency change; an existing skill enters the same loop from its text.

Level 11: the example skill taken through the whole method

This level answers: what does the full design look like for the test-driven-development skill, and how will its numbers be read?

Objective. The sentence from Level 5: correct change, proven by the agent's own tests.

Points. The seven core points C1 to C7 from Level 6, the six secondary test-quality rules, the persuasion block as an adherence device, and the session hook as a loading policy.

Graders. The seven code checks from Level 7 over the harness log, plus a loaded-or-not flag, plus the effect measures from Level 8: hidden tests, one label per agent-written test anchored to runs against the old code and planted faults, honesty, cost and stability.

Tasks. Twenty real Python repository tasks drawn from a benchmark refreshed after the models' training cutoff: 12 bug fixes and 8 features, each with hidden tests, a gold patch under 150 lines, and a test suite that runs in under two minutes. Bug fixes are there because C6 only applies to them; features are there because C1 and C7 show most on new behaviour.

Arms.

• P0, baseline. Nothing.

• P1, the skill as installed. Plugin and session hook, with the task text naming the skill.

• P2, the skill body only. No hook. P1 minus P2 is the price and the effect of the hook.

• P3, the alternative. Code first, tests after, then verify the tests fail on the old code. P3 against P1 asks whether the idea is test-first or verify-your-tests.

• P4, a skill written from P0's failures. Points from what the baseline actually got wrong on these tasks, which may or may not be the seven doctrine points.

• P5, that skill cut by ablation. Only the points that moved the effect.

• P6, the cut skill delivered on demand. Not in the listing, not injected; the agent is told the command that prints it. Same text, zero footprint, so P6 against P5 is the price of the listing.

• P7, the length control. Unrelated sensible text, same token count as P2.

Three runs per task for P0, P1 and P5, two for the rest, ablation of P4's points on a ten-task subset, and the P0, P1 and P5 row repeated on a second, cheaper model: about 600 runs, with a money ceiling set before the first one.

How the numbers are read. The reading rules are fixed before the first run, which is what stops the result from being argued into shape afterwards:

• P1 adherence low: the text does not work on this model, whatever its reputation. Nothing about the ideas is known yet.

• P1 adherence high, outcome and share of critical tests no better than P0: the agent did test-driven development and the code was not better for it. The seven points are doctrine that this model does not need.

• P1 better than P0 but not better than P7: the gain was the extra tokens, not the ideas.

• P1 and P3 equal: writing tests first is not the idea that matters; verifying them is. The skill can lose its Iron Law.

• P2 equal to P1: the hook pays for nothing. P2 worse than P1: the hook is the only reason the body was followed, which is a statement about the text.

• P5 equal to P1 with fewer points, or P6 equal to P5 with no listing: the same effect is available without the per-turn cost from Level 1.

• Any arm's gain is read next to its tokens and turns; an arm that wins by a point and triples the bill is reported as exactly that.

The design is the part a reader can take today. The numbers it produces belong to the model and the month they are run in; they are not a property of the file.

Carry forward from Level 11:

• One objective, eleven labelled points, twelve graders, eight arms, one reading rule per comparison.

• The reading rules are written before the run.

• The numbers are per model and per date; the design is reusable.

The tools and approaches

What exists today for each part of the method, as observed rather than endorsed.

• Headless agent runs. Role: produce the harness log. Gives: a stream of events with every tool call, file write, command and token count per run. Limit: format differs per agent; the graders are written once per runner.

• claude plugin eval. Role: run a case set against a plugin in fresh sessions. Gives: repeated runs per case, a with-and-without ablation mode that reports the per-case delta, deterministic graders such as regex, tool used and file exists alongside a model judge, a model flag for the per-model row, and cost in dollars. Limit: variance has to be computed from the per-run scores (Anthropic, Claude Code plugin evals documentation).

• Skill usage reports. Role: show which installed skills are ever used. Gives: per-skill invocation counts and last use. Limit: a used skill can still be a loss.

• Live coding benchmarks. Role: the task source with hidden tests. Gives: real repository tasks collected after model training cutoffs, with fail-to-pass and pass-to-pass tests (SWE-bench-Live, arXiv 2505.23419; SWE-rebench, arXiv 2505.20411). Limit: Python-heavy, and environments take work to build.

• Length-matched controls. Role: separate ideas from tokens. Gives: the arm that shows whether length alone moved the result. Limit: the control text must be sensible and unrelated, or it tests something else.

• Mutation tools and test judges. Role: anchor the per-test quality label to evidence. Gives: planted faults from mutation tools, then a model judge with a rubric for the tests that survive. Limit: coverage and mutation scores are unreliable on code that is itself buggy (arXiv 2607.22880), and a judge is an opinion until checked against hand labels.

• Skill scanners. Role: catch known-bad patterns in shared skills. Gives: a reported 86.7% precision and 82.5% recall on a labelled set (Agent Skills in the Wild, arXiv 2601.10338). Limit: five products were bypassed in under an hour in a published test; a scan is not a review.

• On-demand instruction libraries. Role: zero footprint until used. Gives: plain text files printed by a command the agent is told about, with no listing in the prompt. Limit: I found no published study of how often an agent finds the right file unprompted.

A decision guide

• Should I install this shared skill? Use: extract its points first. Then check: if no points, do not install; if points, run baseline, skill and length control on a few dozen of your tasks and read the pair of scores.

• Should I write a skill for this task? Use: run the baseline first and read the failures. Then check: if the failures repeat, the points are the failures; if they do not, you need a better task description, not a skill.

• My skill scores high on adherence and the outcome is flat. Use: ablation, one point at a time. Then check: keep only the points whose removal lowers the effect; the rest is doctrine.

• My skill scores low on adherence. Use: rewrite the wording, shorten it, put the trigger condition in the first line, match the examples to the agent's language. Then check: re-run adherence only; do not touch the points until the text is followed.

• The skill helps and costs far more. Use: cut secondary points and persuasion text first. Then check: effect must hold at the lower word count, or the trade is a real one and someone has to decide it with both numbers.

• The skill makes the agent write more tests. Use: label every test, anchored to runs against the old code and planted faults. Then check: only the count of critical tests, because the published trajectory study found test volume does not track whether the task was solved.

• The skill is in a listing of dozens. Use: name it explicitly in the task, or route it by a rule. Then check: the loaded-or-not flag; an unnamed skill loads about half the time.

• We changed the model. Use: re-run the row. Then check: adherence first, because that is what drifts silently.

• Someone asks whether a skill "works". Use: ask for the model, the version, the baseline and the control. Then check: if any is missing, the answer is "not measured".

Takeaways

1. A skill is text in the context. Nothing in the model changes, and everything known about long prompts applies the moment it loads.

2. Installed and used are different things. On one machine, 81 of 89 listed skills had never been invoked; in published evals the model opens a skill about half the time.

3. Published means formatted. The specification has no test, and the measured default of a shared skill is no gain and more tokens.

4. Tokens went up 72% to 451% in the published tests, and an equally long irrelevant text moved the result about as much as the skill did.

5. One in four scanned marketplace skills has a vulnerability pattern, and scanners were bypassed in under an hour. A shared skill is code from a stranger.

6. A skill is accepted per model and per environment. The same file was 33 points up on one model and 22 down on another, and everyone's context, rules, tools and repositories make a different prompt out of the same skill.

7. "It worked for me" is one run with no baseline and no cost count. It cannot see a 3-point change, and it never sees the bill.

8. Share ideas, not text. A skill's points can be judged; its fifteen hundred words can only be felt. A skill with no extractable point cannot be evaluated, only obeyed.

9. Start with one sentence: what must be better, as something you can measure. Unmeasurable means unstudied, and unstudied means subjective text.

10. A short list of core points, each with a testable hypothesis. Points from baseline failures outrank doctrine.

11. Two metric families: adherence, graded per point from the harness log, and effect, graded on real tasks with hidden checks and one quality label per item the agent produced. Read them as a pair.

12. The verdict comes from the pair: low adherence means fix the text; high adherence and flat effect means drop the idea.

13. Always run a length control. A skill that beats baseline and loses to its own length has no usable ideas in it.

14. Ship text, points, evals and results together, and re-run on every model change. Accuracy fell from 95% to 65% in a month when nobody did.

15. The design is reusable; the numbers expire.

References

• Anthropic. Claude Code documentation: skills; plugin evals. https://code.claude.com/docs/en/skills and https://code.claude.com/docs/en/plugin-evals

• Anthropic. How Anthropic enables self-service data analytics with Claude. claude.com blog, June 2026. https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude

• Agent Skills specification. https://agentskills.io/specification

• GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing. arXiv:2610.02952

• How effective are traditional test criteria at detecting bugs in large language models generated code? arXiv:2609.09315

• Agentic Context Engineering: evolving contexts for self-improving language models. arXiv:2510.04618

• Agent Skills Can Be Harmful. arXiv:2608.11888

• Agent Skills in the Wild: an empirical study of security vulnerabilities at scale. arXiv:2601.10338

• Demystifying Agent Skills. arXiv:2608.14036

• Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study). arXiv:2607.22880

• Do Not Mention This to the User: detecting and understanding malicious agent skills in the wild. arXiv:2602.06547 (USENIX Security 2026)

• Memento-Skills: let agents design agents. arXiv:2603.18743

• Radzymiński, S. Mutation Testing for Agent-Written Code. awesome-testing.com, August 2026. https://www.awesome-testing.com/2026/08/mutation-testing-for-agent-written-code

• Meta-Context Engineering via agentic skill evolution. arXiv:2607.16621

• Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents. arXiv:2602.07900

• Signal or Noise: evaluating agent skills on real coding tasks. arXiv:2608.23067

• Skill Retrieval Augmentation (SRA-Bench): lost in implicit routing. arXiv:2604.24594

• SkillsBench. arXiv:2602.12670

• Spence, S. Measuring Claude Code skill activation with sandboxed evals. https://scottspence.com/posts/measuring-claude-code-skill-activation-with-sandboxed-evals

• superpowers skill collection, obra/superpowers, test-driven-development skill and using-superpowers hook. https://github.com/obra/superpowers

• SWE-bench-Live. arXiv:2505.23419

• SWE-rebench. arXiv:2505.20411

• SWE-Skills-Bench. arXiv:2603.15401

• Trail of Bits via Cloud Security Alliance. AI agent skill scanner bypass research note, June 2026. https://labs.cloudsecurityalliance.org/research/csa-research-note-ai-agent-skill-scanner-bypass-20260610-csa/

• Vercel. AGENTS.md outperforms skills in our agent evals. https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals