So, you want to build a software factory?
文章目录
More and more engineers seem to be investigating software factories.
There are many ways to define “software factory”, but here’s the one we’ve settled on: centralizing your system of agents into one place to build software. This means going from custom coding setups on everyone's machines to a single definition. The same way engineers agree on CI, we should agree on how agents are run.
Why is this important? Better security management, better ROI tracking, and mainly, improving agent efficiency as a team (instead of copying ideas to everyone’s local setups).
Over the past few months, we’ve been improving our own team's agent setup to match this way of working, and bringing our learnings to Warp Factories to help engineers build software factories too.
These are the seven criteria we found to be table-stakes for building a software factory:
1. Factories as Code
First, your team should have a shared configuration for your agents that lives in code. The same way you use Terraform to deploy cloud infrastructure, you use a factory configuration to deploy your software factory.
This gives you the typical advantages of version controlled systems: check in changes with code owners, test changes with rollback support, and review updates as diffs.
Centralization is the main piece though. We're moving away from individual secrets, MCPs, and workflows that live on each person's independent agent setup, and unifying them so the team can agree on how agents should behave. One set of model selections, one secrets / MCPs definition, one dashboard to track cost and quality… everything. This moves engineers away from tweaking their personal setups, and towards proposing changes that the whole team can benefit from.
"I've been doing a lot of tuning to our factory and have been really enjoying it. Getting to spend more time on architectural thinking as an engineer is actually pretty fun." – Taylor Austin, Senior Software Engineer, QuantHub
Of course, this factory configuration is accessible to agents too. Agents can propose diffs to its own configuration based on your feedback, benchmark results on new setups, and self-improvement loops where agents review past runs. We’ll discuss these last two in a bit.
2. Unified Conversation History
While unifying the configuration, you’ll also want to bring together all of your team’s agent conversations in one place. This lets you (and your agents) pull insights out of previous sessions to improve your factory setup.
We’ve spoken to teams zipping up Claude or Codex sessions from their computers and shipping them to S3 buckets. Don’t do this! Ideally, all agent sessions should flow through a single source so that conversation traces are stored together by default. This means hosting agents in the cloud, which has a number of benefits, including AFK automations and computer use verification.
Warp Factories is one implementation of this: all conversation traces are stored together, and transcripts can be self-hosted.
In doing this, you should also preserve a personal inbox to see which agents need your attention vs. your teammates. We started using a factory inbox view to filter this list down to runs that need our attention. This way, you still get the experience of commanding your agents from a control panel while having the underlying data shared across your team.
3. Benchmarks
Once you have all of this conversation data in one place, you should be able to put it to use. A common example is sampling data to benchmark different models to find the best cost-to-performance tradeoff.
Benchmarking means replaying past team conversations from a specific repo state, changing the model selection to see how each compare on a metric you define with an LLM as a judge (correctness, efficiency, etc).
The goal is to find the best model for your team's setup, instead of following global benchmarks that aren't tuned to the tasks your factory is accomplishing. Think SWE-bench, but on your team’s own data (“Me Bench” trademark pending).
Benchmarking should allow you to a) evaluate multiple model configurations against past runs, b) plot the cost to performance tradeoff (say, on a pareto curve), and c) deploy that update as code.
Once live, you need tools to measure whether the expected gains hold in production. Our team uses dashboards that track key quantitative metrics, such as cost per PR and total cycle time, alongside qualitative performance scored via LLM-as-a-judge for criteria like efficiency and code quality. This ensures every configuration change delivers measurable value.
4. Chat From Anywhere
Of course, your factory should be accessible everywhere your team works. This is key for adoption across a team. This means multiplayer prompting in Slack, tagging from task boards like Linear or Jira, or fully automating against a backlog.
You’ll find over time that it's convenient to hop between all of these mediums throughout the lifecycle of a task. A common workflow we've brought to Warp Factories' default setup is to use multiplayer conversations in Slack or Microsoft Teams as the main intake for agents, while also having agents track work on a task board to guard against duplicates and generate weekly summaries of what the team worked on.
You'll also probably want a web view to interact with the chat directly. This becomes a source of truth for what was worked on and the artifacts that were generated (plans, PRs, screen recordings, etc).
5. Artifacts
It should be easy to see all of the outputs an agent has generated, not just its activity. This includes linked issues, generated GitHub PRs (potentially multiple when stacking), and most crucially, end-to-end verification with computer use. These should be tracked alongside your conversation traces and, in the case of plans or visual screenshots, attached to the PR.
6. Computer Use
A primary bottleneck for agents right now is end-to-end verification. This means having an agent provide as much proof as possible at the time of PR to give the team confidence to hit merge without line-by-line scrutiny or, worse, pulling down the change against a local setup.
Computer use verification is the most important part. This means giving the agent tools to click around the app that it built and provide both before-and-after screenshots and screen recordings of the user flow.
In our team's setup (and in Warp Factories), all agents have access to a computer use tool that uses frontier models to navigate apps that were built. It also includes auto-trim enhancements to make these recordings easy to review, like calling out cursor clicks and editing out dead space to cut three or four minutes of agent verification down into a 15-second recording.
On our team, this has dramatically cut review times, especially for projects that are hard to mock out locally. A great example is mobile app development. Since our infra supports multiple operating systems (Linux, macOS, and Windows), we can have agents pull up a native simulator of the app and send a screen recording, so we don't have to build that setup locally in XCode.
7. Agent Debugging Tools
In order to pull insights out of your team's conversations, they need to include the full trace of everything that occurred. This means not just the conversation, but also the tool calls, MCP server access throughout the run, and the generated artifacts. One reason is observability by agents to notice failures and self-improve. Another reason is manual debugging with a visual panel.
In Warp Factories, we’ve been adding baseline tools that we use internally to debug both the conversation trace and agent <> subagent orchestration. The timeline view shows the handoff points between agents and lets you click into a subagent’s work to debug. The chat view also shows every tool call that was fired while the agent worked. If you have ideas on debugging options you’d need, we’re open to feedback.
Bringing it together
If there's one thing to take from this, it's that now is the time to start pulling the learnings out of everyone's local agent sessions and unifying them, so the team can work together on making their agents more efficient.
A software factory is a way to do this, and Warp Factories has been building out the infrastructure to make it easy to build a factory of your own, with self-hosting options for every part of the stack.
If you're interested, you can get early access here: warp.dev/get-started. We're providing up to $10k in credits for early teams!
