好文集ChatGPT Plus¥119

I left Ramp and spent eight months rebuilding the setup I couldn’t live without.

文章目录

You’re sitting there with your agent prompting it 50 times before putting up a PR. The big dawgs like Ramp are handing off changes to their agents and starting on the next task. Anthropic reported that agents spend around 80% of interactive session time waiting for human input. More agents shouldn’t mean spending your entire day helping them run the app, interpret failures, and respond to review feedback.

I was a staff engineer at Ramp when @zachbruggeman, @rahulgs, and @monasticpanic built Inspect. Inspect was the best developer tool I’d ever used; I could design a change, hand it off to Inspect, and work on something else while it was implemented, tested, and fixed.

When I left Ramp to start Arcanist, I still had agents that could write code. What I badly missed was the ability to have my code pass through testing, review, QA without my supervision. At Ramp, the setup around Inspect completely automated that process, sending failures back to the agents at each step.

Moving the agents off of my laptop let them run unattended, but we still needed confidence the changes were correct before they reached customers. We had to build checks the agents could run without us. We iterated on review and QA and fed the results back to agents so they could investigate failures and fix them.

For example, overnight we had a review job get stuck. Devin investigated and wrote a fix. Arcanist caught a bug in the fix, and Devin corrected it before merge. After merge, our QA agent automatically tested the change in staging. Once those checks passed, the change deployed to production. I slept through this entire incident.

文章配图

Agents carry a change from alert to production, with simulation and QA checks before it reaches customers.

Working this way has let the three of us put up over 13,000 PRs in the last 3 months. Give this article to your coding agent and ask it to help you implement whatever pieces apply.

An environment agents can work in

An agent needs to start your application and check its work without asking you to set up its environment. Give it one command that starts the required services, loads test data and sets up credentials. Give it the same checks that run in CI so it can find and fix failures before it opens the PR.

Every agent working concurrently needs its own copy of the application and test data. Otherwise, one agent can overwrite the state another agent is testing. Every extra copy of your application uses more memory. Have each agent start only the services it needs.

At Ramp, one copy of the application would take about 30 GB of RAM on my computer, so several copies would quickly exhaust my max memory. Our app is small enough that we still use local worktrees, but a larger app or more agents running side by side will need more capacity.

Build for cloud agents early so your laptop doesn't limit how many changes you can work on at once. Devin and Inspect are the best agent development environments that I've used for working with agents remotely.

Running agents at volume

We use cheaper models for specific tasks to control costs, then have separate agents check their work. For example, DeepSeek investigates individual potential bugs, and other agents verify whether those bugs can happen. Also, for background work that can be a little slower, we use Ramp Router. Their Flex pricing lets us get lower rates on frontier models in exchange for slower responses.

Checks that keep work moving

Once your agents can work in parallel, your CI needs to keep up with them. 10,000 PRs a day is useless if customers have to wait 4 hours to benefit from those changes. Every failed CI check sends the agent into another loop to revise its work, so slow checks compound.

Reduce the wait by caching dependencies, running tests in parallel, and using faster test runners. We moved from GitHub runners to Peak to improve runner speed. Across 400 PR runs before and after the move, median lint time fell from 85 to 32 seconds, and integration test time fell from 100 to 50 seconds.

文章配图

Faster CI shortens the wait for results. Agents can fix failures sooner and get changes ready to ship.

Fast feedback is important, but you also need to ensure it's reliable. If feedback is inaccurate, then agents can spend their time worrying about the wrong problems. A flaky test can fail even if there's nothing wrong with the code, so an agent will need to figure out if it broke something or the test is the issue. Fix flaky tests and remove any tests that don't catch failures that you actually care about. Otherwise, you’re going to waste time chasing false failures or fixing things that don't matter.

Review that can keep up

If your reviewer reports speculative bugs and lots of minor issues, you have to spend time deciding which findings need attention. Require evidence that a reported bug can happen in practice. Don’t waste time investigating failures that depend on inputs your app never accepts.

We built Arcanist to reproduce bugs in a working copy of your application before it reports them. It only flags bugs serious enough to block a release, and each finding gives your agent evidence to work from. This way, you don't have to validate the bug yourself or worry about false alarms.

You can make Arcanist a required check before you merge so that agents have to resolve critical findings before they ship to production. Arcanist is pretty fast. Across 331 runs, median review time was 6.9 minutes, and 95% of reviews finished within 14.2 minutes.

文章配图

One customer’s median PR-to-merge time fell 60%, from 127 to 50.8 minutes, after making Arcanist a required check.

Getting changes through the merge queue

Even if two PRs are fine on their own, they can still break each other when combined. A merge queue can check changes together before they merge, but testing one PR at a time from the queue was so slow that we dropped it entirely.

After reading "Rethink the Merge Queue," by @reasonmachines we brought the queue back. Their approach tests PRs in groups and merges the entire passing group at once. This way, several agents can get through changes at the same time. We gave Claude the article, and it built our first version in one attempt.

The first version spent too much time retesting changes and repeating database migrations. One PR waited 45 minutes to merge. After we removed the unnecessary retesting and repeated migrations, median time from ready to merge was 2.9 minutes.

Let agents catch and fix things after merge

After merge, have a QA agent test changes in staging before they go to production. Staging lets you validate integrations such as Stripe or AWS that are not feasible to test in the sandbox. If a check fails here, then send it back to the coding agent and repeat QA after the fix. Keep changes out of production until all checks pass.

Our Sentry and Datadog alerts go to Slack. Devin investigates the production failures reported in those alerts and opens fix PRs. Those fixes go through the same review and QA process as any other change.

Ensure the alert channel is limited to problems that actually require investigation or a fix. People mute noisy channels, and muted channels make important failures easier to miss. If an alert does not require intervention, tune it or remove it.

What I still do

I still decide what we build and if what we're building solves the customer's problems. This setup lets agents carry a change through to production so that I can focus on the important decisions.

Spend less time checking your agents’ work

Arcanist builds a working copy of your app with realistic test data. It simulates the workflows your changes affect and only reports critical bugs. It gives your agents evidence that they can use to fix those bugs. This way, you spend less time reproducing issues and deciding if a change is safe to ship.

If you want to work like the big dogs, try Arcanist on your next PR. Use code BIGDAWG50 to get 50% off your first 6 months.

Appendix: the tools and subscriptions behind this setup

Hetzner VPS ($140): Always-on assistant for Slack, merge-queue monitoring, and overnight fixes.

Mac mini: Browser QA, onboarding checks, and the subscription proxy.

If you can’t afford a Mac mini, use a VPS for always-on agents.

Coding subscriptions: $1,590 a month

  • 4x Claude 20x ($200 each)

  • Codex 20x ($200)

  • SuperGrok Heavy ($300)

  • Cursor Ultra ($0 with SuperGrok Heavy)

  • Devin Team ($50)

  • OpenCode Go ($40)

  • Morph Ultra ($200)

Subscription-maxx personal plans while they last. I use CLIProxyAPI (Theo’s guide) to put my accounts behind one sign-in. Team plans make sense when you need a shared environment, but their limits are terrible.

全文完
查看原文