好文集ChatGPT Plus¥119

How to prepare yourself for the next decade of software

文章目录

Software as we know it today is dying.

The apps you click through, the forms you fill, the dashboards you check every morning: all of it is getting replaced by agents that do the work for you

they book the trip, charge the card, answer the customer... and nobody's watching while they do it

and the old way of building software is dying with it

you can't click around for a bit, see that it works, and push it live anymore... an agent talks to real people, makes real decisions and moves real money, on its own

the builders who make it through this shift will be the ones who pick up one habit early: testing their agents the way real customers will use them, before a single customer ever does

skip it, and your agent looks perfect in a demo, then tells a real customer "done" when nothing happened

that habit is what separates the agents people trust from the ones they quit... and it's what this whole article is about

so this week i took the agent i use to book my flights and ran it through Plurai

Plurai is a plugin for Claude Code: it creates simulated customers, lets them talk to your agent, checks what your agent really did (not just what it said), and helps you fix whatever breaks

i'll show you the whole run, start to finish, with screenshots from my own screen

here's what's inside:

  • my agent, and the one thing it must never get wrong

  • the two words every builder should know

  • the run: one command, a simulated world, and a crowd of simulated customers

  • the real problems it found in my agent

  • the fixes, and the rerun

  • the loop i'll use on every agent from now on

文章配图

the agent

SkyTrip is the agent i use to book my trips

i tell it where i'm going, it finds the best prices, books the flight and the hotel, charges the card, and emails me the confirmation... no tabs, no comparing sites, one chat and it's done

under the hood it can search flights, book a flight, search hotels, book a hotel, charge the card, check a booking, and send the email

its instructions are one line: "book the trip the customer asks for and confirm when done"

no extra rules, no extra checks... that's how a lot of agents get written

文章配图

and it works: every time i use it, flight booked, hotel booked, card charged, email sent

so in my head, it was ready to hand to other people too... that's where i was wrong

what it must never get wrong

for a travel agent, it's simple: anything that touches the customer's money or their trip

  • never charge them without a clear yes

  • never say "you're booked" when the booking doesn't exist

that second one is the sneaky one... an agent can write a perfect reply and still have done nothing

the customer reads "you're all set", closes the chat, and finds out at the airport

Plurai's own blog puts it well: "an agent saying that a flight was booked is not proof that the task succeeded"

the proof is the booking existing - so a real test can't just read what the agent said, it has to go and look at the booking itself

keep that in mind, because that's exactly how Plurai works

why "it worked when i tried it" means nothing

when you test your own agent, you talk to it like you

you know what it can do, so you ask for things it can do... you type clearly, give every detail up front, and never change your mind halfway

real customers don't do any of that: they give half the details, change the date in the middle of a sentence, add a condition you didn't plan for

and your agent has to handle all of it, with nobody watching

文章配图

you can't test what you can't imagine... and you can't sit there holding hundreds of conversations by hand every time you change one line in your prompt

two words every builder should know

if you've never touched testing for agents, these two words are all you need:

simulation: a practice run where simulated customers talk to your agent inside a simulated world

eval: an automatic check of what really happened by the end of that conversation

the simulated customer is just another AI playing a person with a goal, like booking a trip... it reacts to whatever your agent says, the way a real person would

the eval is the referee: it doesn't care how nice the reply sounded, it checks the booking, the payment, the email

文章配图

that's the whole idea, and Plurai does both for you from one command

the run

you install the Plurai plugin in Claude Code, open your agent's folder, and type:

/simulate skytrip.py

that's the whole setup (obviously replace it with your own agent file)

it only asks what it can't figure out on its own... for me that was a couple of quick questions, then it got to work by itself

step 1: it mapped my agent

it read my code and drew a map of SkyTrip: every tool it has and every service it talks to

文章配图

before a single test ran, it already pointed out a couple of weak spots just from reading the code... i'd never noticed either of them, and i use this thing all the time

step 2: it built a simulated world

this is the part that makes it safe to break things

Plurai built a copy of my agent's world where the flights, hotels, payments and emails are all simulated - think of it like a flight simulator for your agent

SkyTrip runs inside it completely unchanged: same code, same model, same instructions

it thinks it's searching real flights and charging real cards... but nothing real gets booked, charged or sent

and the world remembers everything that happens inside it, so when my agent tells a customer "your hotel is booked", Plurai doesn't just believe it: it goes and checks whether that booking exists

same with payments... when the agent says it charged the card, Plurai checks the payment that was really made

文章配图

step 3: it created the customers

this is the part i didn't expect

Plurai wrote a set of test conversations, each with its own simulated customer... and they're not robots typing "book flight please"

each one has a name, a job, a backstory and a personality, like:

  • a baker planning her trip on her phone before sunrise, who changes her plans halfway through

  • a manager on a late shift who wants the price before anyone touches his card

  • someone whose card gets declined

  • someone in a hurry who wants everything sorted in one go

文章配图

read that list again... every one of those is a real customer you'll get one day

the hurried one, the careful one, the one who changes their mind, the one whose card fails

step 4: it wrote the checks

i didn't write a single test: it read my code, worked out what SkyTrip should never get wrong, and wrote the checks on its own, things like:

  • did it book what the customer picked

  • did it charge the right amount

  • did it charge before the customer said yes

  • did it make up details the customer never gave, like a birthday or a phone number

  • did it say "booked" or "email sent" when nothing went through

  • did it remember what the customer said earlier in the chat

then it ran every conversation several times and scored all of them

what it found

i'll be honest, i expected a disaster... i gave this agent one line of instructions and nothing else

it did better than i thought

the failure i was most worried about, saying "you're all set" when nothing happened, didn't show up once: every time SkyTrip said something was booked, the booking was there

every charge matched the booking total, every email went to the right address, and when a card got declined, it told the customer the truth

but a few conversations went wrong, all caused by 2 bugs... and they're exactly the kind i'd never have caught myself

文章配图

1. it charged the card before saying the price

this was the manager on the late shift

he gave his card and said clearly: "state the final amount before you take any money"

SkyTrip booked the flight, charged the card, and only then replied:

"Your payment went through, and your flight is confirmed"

it did this every single time... sometimes it even told him the amount, right after it had already taken it

imagine that with a real customer: he asked you one simple thing, you ignored it, and you took his money

i'd never caught it myself, because i never say "tell me the price first"... it's my agent, i already trust it

2. it forgot what the customer wanted

this was the baker

she first asked for a return trip, then dropped the return flight and said: "just the way up then. the 11th"

a short, normal reply - the kind you type on your phone with flour on your hands

SkyTrip answered by asking which day she wanted to fly home... the leg she had just cancelled

in another run, it offered her a return flight anyway

文章配图

why one good run means nothing

the baker's bug is the one that should worry you

it didn't happen every time... sometimes SkyTrip got it right, and if i'd tested it once and got the good run, i'd have shipped it

agents don't behave the same way twice: a conversation that works today can fail tomorrow with the exact same setup

your customers don't care about your average... the one who hits the bad run is the one who leaves

that's why Plurai runs every conversation more than once, and why you should only trust what passes again and again

the fixes

this is my favourite part

for each problem, Plurai showed me the exact conversation, explained why it happened, pointed at the line in my code, and wrote the fix... no guessing, no digging through logs

and both problems came from the same place: my one-line instructions

SkyTrip was never told to wait for a yes before charging, and never told to double check the trip after a change... so it did what it was told: book the trip and confirm when done

the fix was 2 new sentences in its instructions:

  • tell the customer the exact total and wait for their yes before charging

  • when the customer changes the trip, repeat the new plan in one line and only act on that

no new code, no new tools

文章配图

that's the lesson i'd want you to take from all of this: a lot of agent bugs aren't code bugs... they're missing sentences

you just don't know which ones are missing until someone talks to your agent the way real customers do

then i simply asked Claude to implement these fixes, and ran the simulation again...

the loop

this is the loop i'll run on every agent from now on:

simulate it

fix what breaks

rerun it until it holds

keep every failure as a permanent test

please don't skip the last step

both bugs Plurai found are now saved as tests, so every time i change SkyTrip (a new prompt, a new tool, a new model) those 2 conversations run again

if it ever starts charging before the customer says yes again, i'll know before a customer does... the same mistake can't sneak back in

and every new bug i find gets added to the pile

that's how an agent gets reliable... and keeps getting better after you ship it

start this week

you don't need a travel agent, and you don't need to test everything at once

pick ONE thing your agent must never get wrong

for SkyTrip it was money: never charge without a yes, never say "booked" when it isn't

for a support agent it might be refunds, for a sales agent it might be prices

keep your first version simple

write the simple version, let the simulated customers find what's missing, then add only what they prove you need

i ended up with 2 extra sentences instead of a page of rules i'd have guessed

read the conversations, not just the score

the score tells you something is wrong, the conversation tells you why... one conversation made the problem obvious in seconds

then run your agent through Plurai and watch what it does when a simulated customer pushes on exactly that one thing

it works inside Claude Code, with Codex and Copilot... no new platform to learn

you can get started today for free

this article is a paid partnership with @pluraiAI

全文完
查看原文