How to prepare yourself for the next decade of software
文章目录
Software as we know it today is dying.
The apps you click through, the forms you fill, the dashboards you check every morning: all of it is getting replaced by agents that do the work for you
they book the trip, charge the card, answer the customer... and nobody's watching while they do it
and the old way of building software is dying with it
you can't click around for a bit, see that it works, and push it live anymore... an agent talks to real people, makes real decisions and moves real money, on its own
the builders who make it through this shift will be the ones who pick up one habit early: testing their agents the way real customers will use them, before a single customer ever does
skip it, and your agent looks perfect in a demo, then tells a real customer "done" when nothing happened
that habit is what separates the agents people trust from the ones they quit... and it's what this whole article is about
so this week i took the agent i use to book my flights and ran it through Plurai
Plurai is a plugin for Claude Code: it creates simulated customers, lets them talk to your agent, checks what your agent really did (not just what it said), and helps you fix whatever breaks
i'll show you the whole run, start to finish, with screenshots from my own screen
here's what's inside:
-
my agent, and the one thing it must never get wrong
-
the two words every builder should know
-
the run: one command, a simulated world, and a crowd of simulated customers
-
the real problems it found in my agent
-
the fixes, and the rerun
-
the loop i'll use on every agent from now on
the agent
SkyTrip is the agent i use to book my trips
i tell it where i'm going, it finds the best prices, books the flight and the hotel, charges the card, and emails me the confirmation... no tabs, no comparing sites, one chat and it's done
under the hood it can search flights, book a flight, search hotels, book a hotel, charge the card, check a booking, and send the email
its instructions are one line: "book the trip the customer asks for and confirm when done"
no extra rules, no extra checks... that's how a lot of agents get written
and it works: every time i use it, flight booked, hotel booked, card charged, email sent
so in my head, it was ready to hand to other people too... that's where i was wrong
what it must never get wrong
for a travel agent, it's simple: anything that touches the customer's money or their trip
-
never charge them without a clear yes
-
never say "you're booked" when the booking doesn't exist
that second one is the sneaky one... an agent can write a perfect reply and still have done nothing
the customer reads "you're all set", closes the chat, and finds out at the airport
Plurai's own blog puts it well: "an agent saying that a flight was booked is not proof that the task succeeded"
the proof is the booking existing - so a real test can't just read what the agent said, it has to go and look at the booking itself
keep that in mind, because that's exactly how Plurai works
why "it worked when i tried it" means nothing
when you test your own agent, you talk to it like you
you know what it can do, so you ask for things it can do... you type clearly, give every detail up front, and never change your mind halfway
real customers don't do any of that: they give half the details, change the date in the middle of a sentence, add a condition you didn't plan for
and your agent has to handle all of it, with nobody watching
you can't test what you can't imagine... and you can't sit there holding hundreds of conversations by hand every time you change one line in your prompt
two words every builder should know
if you've never touched testing for agents, these two words are all you need:
simulation: a practice run where simulated customers talk to your agent inside a simulated world
eval: an automatic check of what really happened by the end of that conversation
the simulated customer is just another AI playing a person with a goal, like booking a trip... it reacts to whatever your agent says, the way a real person would
the eval is the referee: it doesn't care how nice the reply sounded, it checks the booking, the payment, the email
that's the whole idea, and Plurai does both for you from one command
the run
you install the Plurai plugin in Claude Code, open your agent's folder, and type:
/simulate skytrip.py
that's the whole setup (obviously replace it with your own agent file)
it only asks what it can't figure out on its own... for me that was a couple of quick questions, then it got to work by itself
step 1: it mapped my agent
it read my code and drew a map of SkyTrip: every tool it has and every service it talks to
before a single test ran, it already pointed out a couple of weak spots just from reading the code... i'd never noticed either of them, and i use this thing all the time
step 2: it built a simulated world
this is the part that makes it safe to break things
Plurai built a copy of my agent's world where the flights, hotels, payments and emails are all simulated - think of it like a flight simulator for your agent
SkyTrip runs inside it completely unchanged: same code, same model, same instructions
it thinks it's searching real flights and charging real cards... but nothing real gets booked, charged or sent
and the world remembers everything that happens inside it, so when my agent tells a customer "your hotel is booked", Plurai doesn't just believe it: it goes and checks whether that booking exists
same with payments... when the agent says it charged the card, Plurai checks the payment that was really made
step 3: it created the customers
this is the part i didn't expect
Plurai wrote a set of test conversations, each with its own simulated customer... and they're not robots typing "book flight please"
each one has a name, a job, a backstory and a personality, like:
-
a baker planning her trip on her phone before sunrise, who changes her plans halfway through
-
a manager on a late shift who wants the price before anyone touches his card
-
someone whose card gets declined
-
someone in a hurry who wants everything sorted in one go
read that list again... every one of those is a real customer you'll get one day
the hurried one, the careful one, the one who changes their mind, the one whose card fails
step 4: it wrote the checks
i didn't write a single test: it read my code, worked out what SkyTrip should never get wrong, and wrote the checks on its own, things like:
-
did it book what the customer picked
-
did it charge the right amount
-
did it charge before the customer said yes
-
did it make up details the customer never gave, like a birthday or a phone number
-
did it say "booked" or "email sent" when nothing went through
-
did it remember what the customer said earlier in the chat
then it ran every conversation several times and scored all of them
what it found
i'll be honest, i expected a disaster... i gave this agent one line of instructions and nothing else
it did better than i thought
the failure i was most worried about, saying "you're all set" when nothing happened, didn't show up once: every time SkyTrip said something was booked, the booking was there
every charge matched the booking total, every email went to the right address, and when a card got declined, it told the customer the truth
but a few conversations went wrong, all caused by 2 bugs... and they're exactly the kind i'd never have caught myself
1. it charged the card before saying the price
this was the manager on the late shift
he gave his card and said clearly: "state the final amount before you take any money"
SkyTrip booked the flight, charged the card, and only then replied:
"Your payment went through, and your flight is confirmed"
it did this every single time... sometimes it even told him the amount, right after it had already taken it
imagine that with a real customer: he asked you one simple thing, you ignored it, and you took his money
i'd never caught it myself, because i never say "tell me the price first"... it's my agent, i already trust it
2. it forgot what the customer wanted
this was the baker
she first asked for a return trip, then dropped the return flight and said: "just the way up then. the 11th"
a short, normal reply - the kind you type on your phone with flour on your hands
SkyTrip answered by asking which day she wanted to fly home... the leg she had just cancelled
in another run, it offered her a return flight anyway
why one good run means nothing
the baker's bug is the one that should worry you
it didn't happen every time... sometimes SkyTrip got it right, and if i'd tested it once and got the good run, i'd have shipped it
agents don't behave the same way twice: a conversation that works today can fail tomorrow with the exact same setup
your customers don't care about your average... the one who hits the bad run is the one who leaves
that's why Plurai runs every conversation more than once, and why you should only trust what passes again and again
the fixes
this is my favourite part
for each problem, Plurai showed me the exact conversation, explained why it happened, pointed at the line in my code, and wrote the fix... no guessing, no digging through logs
and both problems came from the same place: my one-line instructions
SkyTrip was never told to wait for a yes before charging, and never told to double check the trip after a change... so it did what it was told: book the trip and confirm when done
the fix was 2 new sentences in its instructions:
-
tell the customer the exact total and wait for their yes before charging
-
when the customer changes the trip, repeat the new plan in one line and only act on that
no new code, no new tools
that's the lesson i'd want you to take from all of this: a lot of agent bugs aren't code bugs... they're missing sentences
you just don't know which ones are missing until someone talks to your agent the way real customers do
then i simply asked Claude to implement these fixes, and ran the simulation again...
the loop
this is the loop i'll run on every agent from now on:
simulate it
fix what breaks
rerun it until it holds
keep every failure as a permanent test
please don't skip the last step
both bugs Plurai found are now saved as tests, so every time i change SkyTrip (a new prompt, a new tool, a new model) those 2 conversations run again
if it ever starts charging before the customer says yes again, i'll know before a customer does... the same mistake can't sneak back in
and every new bug i find gets added to the pile
that's how an agent gets reliable... and keeps getting better after you ship it
start this week
you don't need a travel agent, and you don't need to test everything at once
pick ONE thing your agent must never get wrong
for SkyTrip it was money: never charge without a yes, never say "booked" when it isn't
for a support agent it might be refunds, for a sales agent it might be prices
keep your first version simple
write the simple version, let the simulated customers find what's missing, then add only what they prove you need
i ended up with 2 extra sentences instead of a page of rules i'd have guessed
read the conversations, not just the score
the score tells you something is wrong, the conversation tells you why... one conversation made the problem obvious in seconds
then run your agent through Plurai and watch what it does when a simulated customer pushes on exactly that one thing
it works inside Claude Code, with Codex and Copilot... no new platform to learn
you can get started today for free
this article is a paid partnership with @pluraiAI













