We’re Thinking Too Small About Synthetic Data

Synthetic data is much more than large scale inference or pre-training data mixture design. It's a deep data research problem, and much of it is still wide open.

What are we even synthesizing today?

It’s easy to think of synthetic data as example generation. A typical process might be to give a model some prompts or templates and output a larger training corpus. When producing enough examples was the bottleneck, a lot of the work was fun engineering stuff, like spending days figuring out how to share prompt prefixes and reuse the KV cache to squeeze more examples out of the same GPUs.

Things are looking different now. We’re training agents to handle whole workflows. We’re designing the whole system that creates those training experiences. That means we need to build usable tasks and environments, along with verifiers that judge success fairly. Otherwise, cheap data just ends up consuming expensive training compute with little to no learning value.

The research problem goes beyond generation and data mixture design. I’ve been thinking about simulation vs distillation for synthetic data, as treating all of this as distillation overlooks what generators contribute.

The generator itself is a research problem.

We don't necessarily need a generative model to generate a world

And you know what… not every part of the generator needs an LLM. Python code still works! And maths and physics exist for a reason. State-transition models and discrete-event simulation can generate a world’s history without an LLM, with real use as the prior for the relationships and situations we choose to model.

Take the customer service agent training example I wrote about earlier.

The agent’s starting world is the end state of this simulated history.

It’s tempting to generate the tickets, emails, and refund records separately, then patch the inconsistencies afterward. For example, we might manually change a payment record from $100 to $70 in database, but overlook the email that still says $100. When scenarios get complex, it's even more expensive and unreliable to check after the fact.

I’d rather generate the payment record from the events that led to it. Things like ticket statuses, balances, and past events would live in a shared state, updated by explicit rules: sₖ₊₁ = F(sₖ, eₖ), where sₖ is the current world state, eₖ the event, and F the transition rule.

One way to build this is a discrete-event simulator that advances the clock from event to event, using state-dependent rules and distributions for ticket arrivals, refunds, and escalation delays. We generate the records from the history it produces, so their relationships are established during generation. This isn’t a novel approach; it has been battle tested in real-world professional domains for decades.

This buys us strong internal consistency.

Humans and synthetic systems better together

While simulation can give us consistency by design, we still have to decide whether its assumptions reflect the real world and which situations are worth learning from. Real human knowledge gives us a prior for those choices:

• Distribution design: human knowledge helps us ground the right set of users and workflows/capabilities to train for.

• Realism grounding: use examples from people doing the work to capture the constraints and details our simulation misses.

• Human review: domain experts can catch answers that pass our checks but would cause trouble in practice, before we reinforce them in training.

Designing the right training distribution

With a limited training budget, we need to choose whose problems we want it to handle and which capabilities to develop.

Imagine we’re building a personal agent, but focusing on “pushing the frontier” with PhD-level intelligence. This could mean making all those users patient and thoughtful about what they want. We could end up benchmaxxing and still miss what a real, impatient human needs when they say “my wifi is down and my meeting starts in five minutes”. The agent might start diagnosing the router, but the simple, often right answer may just be to switch to a hotspot.

Then, on task distribution. If the agent already handles routine status lookups well, I’d give it fewer of those and more of the escalation decisions it keeps getting wrong. Status lookups might be more common in real use, but it doesn’t mean they deserve more of the training budget.

Essentially, designing the distribution is deciding who to train for and where it needs more practice. Real use is the prior; learning value reweights it.

Pushing realism in content “texture”

There’s content (is the information correct and coherent across everything the model sees?) and there’s texture (does it look and behave like the messy mix of artifacts humans actually work with?).

We’ve seen a steady stream of great knowledge work benchmarks recently. DAYJOB does a good job simulating complex professional work from short requests. AA-Briefcase puts tasks inside multi-week business projects. These prompt contents are getting much more realistic.

But realistic prompts + synthetic-looking files isn’t realistic work yet. Imagine a legal agent getting 10 PDFs from different stakeholders, all in the same template. The content might be detailed and diverse, but the texture of those files is a dead giveaway of a fake world.

To get this texture right, we will need much real world references from experts who do the work. Those references will be the source to ground our generators with realism.

Augmenting machine checks with human judgment

Automated checks can cover a lot of quality issues, yet we’ll still keep human in the loop to spot problems those checks miss. In the customer support case, an escalation request might get all the facts right, but somehow the customer sounds perfectly calm after several angry messages. A human reviewer could read the thread and think, “wait, this feels off”, even after all factual checks have passed.

Sometimes we’re rewarding the wrong thing in the first place. In Anthropic’s August 2026 report, a model gamed an honesty reward by piling on disclaimers. They already had humans reviewing flagged environments, but generation was outpacing their ability to check and fix them. Figuring out how to keep up with HITL is part of the research too, and a key part of making training safer.

Looking ahead to adaptive curriculum design

All of these are parts of the generator we should be researching and improving as a system. We should judge it by training lift at a fixed budget.

“any static procedure for synthetically generating data will quickly become outstripped.” — David Silver and Richard Sutton, Welcome to the Era of Experience

I believe the combo of physics- and math-grounded simulation + LLMs + human expertise can go a very long way.

Once we've encapsulated the generator as a research problem, agents could start improving it and using it to train their next versions, and repeat. This could become a foundation for RSI.

We're still very early.