Harness Engineering: How to Build AI Agents That Actually Work
文章目录
Most people are trying to fix AI agents at the wrong layer.
When an agent fails, they rewrite the prompt. When it fails again, they add more instructions, switch models, increase the context window, or connect another tool.
Then the same problems come back.
The agent forgets an important decision. It uses the wrong tool. It loses track of what happened three steps earlier. It claims the task is finished without checking the result. It retries the same failed action until the budget is gone.
The problem is not always the model.
The problem is the environment around it.
That environment is the harness.
Harness Engineering is the practice of building the system around a model that decides what it can see, what it can do, what it remembers, what counts as success, and what happens when something fails.
A better prompt can improve one response.
A better harness improves every run.
Follow my Substack for more practical breakdowns of AI agents, automation and production systems: substack.com/@lunarresearcher
1. The Model Is Not the Agent
A model can reason, generate, compare and choose.
That does not make it a reliable agent.
A real agent also needs to find the right context, use tools, preserve state, respect permissions, verify its own work and recover when the environment behaves differently than expected.
The model is only the reasoning engine.
The harness is everything that turns that reasoning into real execution.
USER REQUEST
|
v
+-----------------------------+
| HARNESS |
| |
| contract context |
| tools state |
| policy verification |
| traces recovery |
+-----------------------------+
|
v
MODEL
|
v
REAL ENVIRONMENT
Put the same model inside a chat box and it answers questions.
Put it inside a repository with terminal access, tests, browser tools, project memory, controlled permissions and a review loop and it can complete real work.
The model did not change.
The harness did.
2. Turn Every Request Into a Contract
Natural language is flexible.
Autonomous execution should not be.
A request like:
Improve the onboarding flow.
is fine when a human is sitting next to the model.
It is terrible as a production instruction.
Before the agent acts, turn the request into a bounded task contract.
objective: reduce onboarding drop-off
inputs:
- product brief
- analytics data
- repository
constraints:
- preserve authentication
- do not change database schema
- preserve current mobile behavior
deliverable:
- reviewable pull request
done_when:
- tests pass
- analytics event fires correctly
- desktop flow passes review
- mobile flow passes review
approval_required:
- production deployment
The important part is done_when.
Without it, the agent can solve a slightly easier version of the problem and still confidently say the task is complete.
With it, completion becomes measurable.
The agent should not ask:
What should I do next?
It should ask:
What action moves the current environment closer to the contracted outcome?
That is a much stronger loop.
3. Give the Agent a Map, Not a Giant Context Window
A common reaction to agent mistakes is to give the model more context.
More documentation.
More conversation history.
More files.
More tool output.
Eventually the agent receives everything and understands less.
Context is not storage.
It is an attention budget.
Instead of dumping the entire project into every run, give the agent a small map of where useful information lives.
PROJECT MAP
product rules -> docs/product/
architecture -> docs/architecture.md
frontend -> apps/web/
backend -> services/api/
tests -> tests/
commands -> docs/commands.md
security -> docs/security.md
Then expand only when needed.
TASK
|
v
PROJECT MAP
|
v
RELEVANT SYSTEM
|
v
EXACT FILES
|
v
LOCAL INSTRUCTIONS
The source material describes this as progressive disclosure: the harness should load more information because the task needs it, not simply because the information exists.
The goal is not maximum context.
The goal is maximum useful signal.
4. Put a Gateway Between the Model and Its Tools
A model with twenty tools is not automatically twenty times more capable.
It may just have twenty more ways to fail.
Every tool should have a contract.
TOOL: edit_file
INPUTS
path
patch
PRECONDITIONS
path exists
path is inside workspace
SUCCESS
patch applied
diff returned
FAILURE
structured error
no partial overwrite
RISK
reversible
Then the execution path becomes:
MODEL PROPOSES
|
v
GATEWAY VALIDATES
|
v
POLICY AUTHORIZES
|
v
TOOL EXECUTES
|
v
HARNESS RECORDS RESULT
The model decides what action it wants.
The harness decides whether that action is valid, permitted and safe.
That distinction becomes critical when tools can send messages, modify production, spend money or delete data.
A good tool gateway can also add timeouts, validate arguments, restrict file paths, normalize errors and make retries safe.
Good tools reduce the number of things the model has to guess.
5. Move Memory Outside the Conversation
The conversation should not be the system of record.
Long-running agents eventually hit context limits, crash, restart, or hand work to another session.
If every important decision exists only inside the transcript, the workflow is fragile.
Store durable state separately.
{
"task_id": "feature_042",
"status": "verifying",
"current_step": "mobile_check",
"completed": [
"implementation",
"unit_tests",
"desktop_check"
],
"decisions": [
"reuse existing export endpoint",
"preserve current date format"
],
"artifacts": [
"export.csv",
"desktop-after.png"
],
"open_risks": [
"mobile toolbar may overflow"
],
"next_action": "render mobile viewport"
}
A useful system separates memory into four categories:
FACTS
stable knowledge
DECISIONS
what was chosen and why
STATE
where the current run is
LESSONS
failures that should affect future runs
The next agent session should inherit the state of the work, not a compressed story about the previous conversation.
6. Make Evidence the Gate to Completion
An agent saying "done" is not proof that the job is done.
It is another model output.
The harness needs observable evidence.
CLAIM EVIDENCE
"bug is fixed" failing test now passes
"page works" browser flow completed
"data is correct" values match source
"migration is safe" dry run + rollback pass
"task is complete" every acceptance check passes
Use deterministic checks first.
syntax
|
v
types
|
v
focused tests
|
v
integration tests
|
v
visual / semantic review
|
v
human approval
Do not ask another model to answer something a compiler, test, schema or database query can prove.
Use models for judgment.
Use deterministic systems for facts.
The model creates the artifact.
The environment creates evidence about the artifact.
The harness decides whether the evidence is sufficient.
7. Separate the Builder From the Verifier
There is another problem with self-review.
The agent that created the mistake often carries the same assumptions into the review.
A stronger architecture separates the worker and the verifier.
BUILDER
|
v
creates candidate
|
v
VERIFIER
|
+-- checks contract
+-- searches for missing cases
+-- tests unsupported claims
+-- tries to break result
|
+------ PASS ------> ACCEPT
|
+------ FAIL ------> RETURN EVIDENCE
The verifier should not ask:
Does this look good?
It should ask:
What would make this unacceptable?
That changes review from confirmation into attempted disproof.
The source material explicitly recommends giving verification its own rejection criteria and enough independence to challenge the assumptions that produced the first result.
8. Move Permissions Outside the Model
Some rules should never depend on the model remembering them.
never publish without approval
never expose secrets
never exceed the spend limit
never write outside the workspace
never claim tests passed unless they ran
These are not prompt suggestions.
They are policy.
A simple permission ladder:
LOW RISK
read
search
inspect
-> automatic
REVERSIBLE
edit workspace
run tests
create draft
-> automatic + trace
EXTERNAL EFFECT
send
deploy
purchase
-> approval required
IRREVERSIBLE / SENSITIVE
delete data
rotate credentials
publish globally
-> hard gate or prohibited
The stronger the consequence, the stronger the control.
The model can recommend the action.
The harness authorizes it.
The tool executes it.
Autonomy is not the absence of control.
It is freedom inside an enforced boundary.
9. Stop Retrying Blindly
One of the worst recovery policies is:
Something failed. Try again.
If nothing changes, the system is simply paying to reproduce the same failure.
Failures should be classified first.
TOOL TIMEOUT
-> retry with backoff
INVALID ARGUMENTS
-> repair tool call
MISSING CONTEXT
-> retrieve missing source
FAILED TEST
-> inspect failing behavior
PERMISSION DENIED
-> request approval
CONFLICTING REQUIREMENTS
-> escalate
UNCHANGED REPEATED FAILURE
-> stop
A useful agent loop looks like this:
OBSERVE
|
v
DECIDE
|
v
ACT
|
v
MEASURE
|
+---- ACCEPT
|
+---- REPAIR
|
+---- ESCALATE
|
+---- STOP
Every loop should have limits on attempts, time, spend and destructive scope.
A reliable agent needs to know how to continue.
It also needs to know when another attempt is no longer worth it.
10. Turn Repeated Instructions Into Infrastructure
Suppose the prompt contains:
Always run the formatter.
That rule is stronger if the formatter runs automatically.
Suppose the instructions say:
UI code may not directly access the database.
That is stronger as an architecture test that fails when the rule is broken.
The progression looks like this:
EXPLANATION
|
v
CHECKLIST
|
v
TEMPLATE
|
v
AUTOMATED CHECK
|
v
ENFORCED POLICY
The prompt should explain judgment.
The harness should enforce invariants.
Every recurring mistake should move a little further down this ladder.
Eventually the model no longer needs to remember the lesson.
The environment remembers it for the model.
11. Record the Run
A perfect final artifact can hide a terrible execution path.
Maybe the agent accessed the wrong source.
Maybe it ignored a failed command.
Maybe it repeated an external action twice.
Maybe it spent ten times the expected budget.
Maybe it got the right answer for the wrong reason.
Record enough information to reconstruct what happened.
09:14 task contract created
09:15 architecture.md loaded
09:17 checkout.ts edited
09:18 focused test failed
09:21 implementation repaired
09:22 focused test passed
09:24 integration test passed
09:25 deployment blocked: approval required
Useful traces include context sources, tool calls, state changes, verification results, retry reasons, approval decisions, cost and latency.
The point is not to collect logs for fun.
The point is to make failure local.
If step 18 breaks, you should be able to repair step 18.
You should not need to replay the entire run.
12. Give Every Run a Receipt
Do not force the human to review a forty-message transcript.
Compile the result into a small receipt.
OBJECTIVE
Fix duplicate coupon application.
CHANGED
checkout validation
regression test
VERIFIED
lint passed
unit tests passed
integration test passed
NOT VERIFIED
production payment provider
RISKS
legacy mobile client unavailable
APPROVAL NEEDED
deploy to staging
This is not a summary of what the model claims happened.
It is a summary of what the harness can prove happened.
That distinction makes the receipt useful for review, handoffs and future agent sessions.
13. Make Every Failure Improve the Harness
Most teams fix the failed output.
The better approach is to fix the system that allowed the failure.
MISSING CONTEXT
-> improve project map
WRONG TOOL
-> improve routing or tool contract
BAD OUTPUT
-> add validator
REPEATED LOOP
-> add retry cap
UNSAFE ACTION
-> add permission gate
LOST DECISION
-> persist state
UNKNOWN FAILURE
-> improve tracing
This is where harness engineering starts to compound.
One repaired output helps one run.
One repaired harness improves every run after it.
The best agent systems get more reliable because mistakes leave infrastructure behind.
14. Start With the Smallest Useful Harness
You do not need a huge orchestration platform to start.
Build in layers.
LEVEL 0
prompt
model
LEVEL 1
task contract
project map
tools
LEVEL 2
structured state
verification
bounded loop
LEVEL 3
permissions
traces
recovery
human gates
A short research task may only need a prompt and one review.
A six-hour coding task with file access, network access and deployment capability needs much more.
Add complexity when the failure surface earns it.
Not because agent architecture looks cool.
The Harness Engineering Checklist
Before giving an agent meaningful autonomy, ask:
[ ] Is success defined before execution?
[ ] Can the agent find the right context
without loading everything?
[ ] Does every tool have a clear purpose,
schema and failure state?
[ ] Are important decisions stored
outside the conversation?
[ ] Does completion require evidence?
[ ] Are risky actions protected by policy?
[ ] Does every loop have a retry limit?
[ ] Can the run resume after interruption?
[ ] Can you reconstruct every important action?
[ ] Does failure improve a rule, tool,
test, map or permission?
[ ] Can the final change be rolled back?
If several answers are no, a stronger model will not automatically make the agent reliable.
It may simply make the failure faster and more expensive.
The Real Shift
Prompt engineering asks:
What should I tell the model?
Context engineering asks:
What should the model know right now?
Harness engineering asks:
What system lets the model act, verify its work, recover from failure and operate safely?
PROMPT
-> instruction
CONTEXT
-> working view
HARNESS
-> operating environment
LOOP
-> local correction
GRAPH
-> coordination
Models will keep changing.
The durable advantage lives around them.
Your contracts get better.
Your tools get better.
Your tests get better.
Your state gets cleaner.
Your permissions get safer.
Your recovery logic gets smarter.
Your failures turn into infrastructure.
That is how capable models become reliable agents.
That is Harness Engineering.
If You Made It This Far
Bookmark this guide.
Follow me on X: x.com/0xjmori
Subscribe to my Substack: substack.com/@lunarresearcher
Send this article to someone who is still trying to fix every agent failure with a longer prompt.
