Software Factories: It’s All About the Data (Architecture)

If you’re grappling with whether to adopt a software factory for your company, consider this: software factories produce software, but they also produce an increasingly detailed record of how the software got made.

If Codex and Claude are already so capable, why not just give them directly to every developer and let everyone work with their own agents? You should probably do that too. But if a meaningful share of your company’s engineering work is going to happen through agents, there is far too much data being generated along the way to leave it scattered across thousands of individual sessions.

Think about it. Code has never been more disposable. You (or anyone else) can regenerate increasingly large amounts of it with the help of a coding assistant and an adequate prompt. So what is defensible then? The speed at which your company works, how quickly you can deliver quality code that matches customer expectations, and how quickly you can continue to innovate.

Every agent run contains information about where the bottlenecks were, why agents got stuck, who had to intervene, and what additional context helped. Most of these signals aren’t particularly valuable in isolation, but across thousands of runs managed by a central software factory they start to describe the production system itself (and therefore your company).

In a sense, software factories are data factories. Underneath them are data pipelines that enable a self-improvement loop. Here’s a look at some of the emerging data architectures behind them.

Every Turn Generates Data

Agentic software naturally generates much richer traces than conventional applications because a single request can produce hundreds of intermediate operations before reaching an output.

Factory provides a good example. Droid can export OpenTelemetry metrics and, optionally, message-content trace spans to a company’s own collector, while Factory’s internal tracing separately powers its hosted Analytics API. The exported data can include model and tool activity alongside usage, duration, and other attributes, depending on the telemetry configuration. Customer-side spans are sent only to the customer’s configured collector and never to Factory’s collector.

With trace export enabled, if an agent reads fifteen files before changing one, calls an MCP server four times, retries a test, and eventually needs human intervention, the company can preserve that sequence rather than only the resulting commit.

At the scale of one run, most of this looks like debugging info. Across thousands of runs, it can tell you the shape of your software process. The factory can find repositories that consistently require longer runs, compare models against similar work, or discover which tools tend to appear before successful outcomes.

The Factory Generates Data About the Data

You can’t necessarily tell from trace data or basic telemetry whether what happened was good or not. Evals creates another data layer around the factory.

Warp is building this directly into its architecture. Historical agent traces can be evaluated by scorers for dimensions such as task compliance, efficiency, and quality. LLM-as-a-judge turns an execution trace and its artifacts into another structured observation about the work.

This approach works even better when the evals run across a large batch. Warp’s self-improvement architecture accumulates scored runs so another agent can look for patterns across successes and failures. A model might struggle with a particular class of migrations, or agents working in one repository might consistently require more human intervention.

You can then take what you learn from those evals to improve your factory. Warp makes this easy because the entire factory is in a git repo that is version controlled and modifiable by an agent.

Production History As an Eval Dataset

The above accumulated data can become an eval dataset in and of itself. Instead of changing a model or skill and waiting to see what happens in production, historical work can be run through the new configuration.

Warp Factory Benchmarks are built around this idea. Past factory runs can be replayed from their original state against different models, harnesses, skills, prompts, and configurations, then evaluated using the same scoring infrastructure.

The same approach can apply throughout the factory. Rewrite a skill and replay the tasks that used it. Change the context strategy and see whether historical tasks require fewer calls. Add another verification step and measure whether the resulting implementations actually improve.

Expanding the Dataset to the Entire SDLC

Port approaches the data problem from a different angle. Its starting point isn’t the execution trace of a coding agent, but a data model of the engineering organization.

Port’s Context Lake ingests information from systems across the SDLC and maps it into a common semantic model. Blueprints define entities such as services, teams, environments, deployments, and incidents, while relationships connect those entities into a model of how the organization fits together.

Port can know which service that repository belongs to, which team owns it, where the service runs, and which deployments or incidents relate to it.

The real benefit comes from the join of this data with the PR. An agent run can be connected to the PR it produced, the PR to a deployment, and the deployment to what happened in production. Now the company can ask questions about that data to improve the end-to-end release process.

Port’s Engineering Intelligence already connects metrics across delivery, reliability, standards, developer experience, and AI adoption back to the services and teams in its Context Lake. Agents can analyze changes in those metrics, while scorecards and workflows can turn findings into action.

Checkpoints Within the Agent Loop

Most agent infrastructure is built around observing execution, which still leaves us reconstructing the actual state of the work from prompts, tool calls, model output, and other pieces of the trace.

At ThruWire, we’re focused on an architecture built around AI checkpoints. The idea is to put durable checkpoints around the agent loop rather than trying to prescribe everything that happens inside it. Between checkpoints, the model gets freedom to work to its full capability. It can reason, call tools, and change direction. At a checkpoint, though, it has to show its work in a form the system can understand.

The requirement is to present ThruWire with the right-shaped artifacts at the right time, in a form it can validate and trust. ThruWire does this with goal-defining blocks, provable relationships between artifacts with attestation, and server-side schema enforcement and validation (now Jev-powered). Put together, it’s not only a receipt system but a self-describing obstacle course for the agent.

There is a powerful data implication to this architecture. Every checkpoint takes something that existed implicitly inside the agent loop and materializes it into the data model. The same semantic processing we use to govern the work also provides an invaluable data stream that customers can explore through data visualizations.

Instead of reconstructing these states from a giant trace later, the system generates structured data about the work while the work is happening. The more work that passes through checkpoints, the richer that data becomes.

Another Sensor Inside the Software Factory

Adding a checkpoint-based data source to factories can be transformational. Factories are already becoming heavily instrumented, with workers generating execution traces while Git, CI, tool integrations, and LLM-as-a-judge add more observations around them. Checkpoints introduce another observation surface: structured observations of the work itself as it moves through the production line.

Consider a worker implementing a feature. Its trace might show that it read 23 files, called three tools, changed eight files, and ran the tests twice. A checkpoint could show that four requirements are satisfied, one remains uncertain, and verification is still needed against a particular dependency. The software factory could use this data to continually reshape the quality of the work, making checkpoints part of the self-improving loop.

Across thousands of runs, those observations start to reveal patterns in the work that aren’t visible in execution traces alone. The factory can find planning states that frequently lead to implementation retries, identify requirements that repeatedly become uncertain, and compare checkpoint assessments against later human intervention, verification failures, or production outcomes.

The same data can feed LLM-as-a-judge and factory optimization. Judges can operate over intermediate states of the work instead of reconstructing everything from execution traces, while patterns across checkpoints can suggest changes to planning, context, workers, or verification.

In other words, we’re building a dataset of the work that keeps today's workers on the rails and also provides feedback to make tomorrow’s factory better.

The Factory Becomes a Data Moat

There’s another implication to accumulating all of this data. Two companies might use the same models, the same coding agents, and even many of the same tools, but their factories can still become very different over time.

Every run adds proprietary information about how software gets built inside that particular company. Which prompts work for its codebase, which tools agents need for certain tasks, where they tend to get stuck, what kinds of changes require human intervention, and which approaches ultimately make it through verification and into production.

As that history accumulates, the factory can start adapting around it. That makes the history of the factory potentially valuable in its own right. Each piece of software produced adds data point to a growing chest of learned institutional experience.

Software Factories Are Data Factories

Software development has always produced plenty of artifacts, but much of the process connecting them has been difficult to observe. Software factories change that because the work increasingly passes through systems capable of recording their own operation.

That creates an optimization loop we haven’t really had before. The factory does work and generates data about the work, patterns in that data suggest changes to the factory, and historical work provides the dataset for testing those changes.

We’re still very early in the development of software factories, and the architectures for capturing and using this data are only beginning to take shape. As more software moves through these systems, we’ll learn much more about what data matters.