Code Builds the World, Diffusion Paints Reality: AgentGarten Gives Agents a Real-Time Arena to Evolve While Playing
Introduction: When “Agents” Stop Being Told and Start Growing
Imagine an AI that can actually stay in a world: it pushes a partition, and the corridor seals shut; it drags over a ramp, and from then on high walls have a way over. Every action changes the world, and the changed world becomes the starting point of its next decision. Now imagine that world is not a hand-crafted game demo, but “written by code and painted in real time by a diffusion model.” That’s exactly what AgentGarten, released by the MirroS team, offers—a real-time “arena for evolving agents.”
On October 8, 2026, MirroS (in collaboration with Tsinghua and Peking University) open-sourced AgentGarten: the project page, a technical report, full code on GitHub, and a detailed blog are all public. The core idea is deceptively simple but conceptually sharp: “Code builds the world, diffusion paints reality.” Rules and dynamics live in executable code; visual feedback is generated by a shared neural renderer. The two are coupled into a real-time loop at more than 30 fps, letting agents interact with a world where “every step you take has a consequence you can see.”
And the most striking result is the learning efficiency gap: in a classic hide-and-seek experiment, the hider learns on round 4 to drag a partition as a shield, the seeker on round 10 learns to cross a wall with a ramp—whereas a conventional reinforcement learning counterpart needs millions of rounds. AgentGarten achieves this because it doesn’t just train an agent with rewards; it “evolves” agents through a world.
In this article I want to dissect AgentGarten at three layers: first, the architectural essence of “code world + shared neural renderer”; second, the two core engineering tricks that make real-time interaction possible (geometry-conditioned rendering and Adversarial Forcing); third, the “playbook” loop that lets experience be inherited across generations. I’ll end with what it means for the future of agent training—a direction where “agents keep evolving through interactive experience” is no longer a slogan but a working mechanism.
Part One: “Code Builds the World, Diffusion Paints Reality”—the Architecture Split that Solves the Dilemma
Before AgentGarten, getting a “faithful + realistic” environment was a painful trade-off. Let me lay out the dilemma it faces and breaks.
On one side, conventional simulators (Box2D, MuJoCo, game engines) hold explicit state and rules: you know exactly the position and velocity of every object, and physics is computed deterministically by code. But building “visually diverse” scenes with them is expensive—every new environment needs bespoke assets, shaders, lighting. Visual realism is hard.
On the other side, video world models produce rich, realistic imagery with ease, but they keep task-relevant state and transition rules implicit—embedded in generated history, not verifiable or controllable. You can’t tell a video model “this door is now closed” with guarantee.
AgentGarten’s answer is a clean separation of responsibilities:
┌────────────────────────────────────────────────────────────────┐
│ AgentGarten design: split "rules" and "vision" │
├────────────────────────────────────────────────────────────────┤
│ ┌─────────────────────┐ export structured ┌────────────┐│
│ │ scene program │ ──── conditions ──────▶ │ shared ││
│ │ (simulator / game │ depth / surface normal │ neural ││
│ │ engine) │ │ renderer ││
│ │ · persistent state │ │ ││
│ │ · code-defined │ ◀──── action ─────── │ diffusion ││
│ │ interaction rules│ │ renderer ││
│ └─────────────────────┘ └─────┬──────┘│
│ ▲ │ │
│ │ action ┌──────────────┐ │ │
│ └────────── │ ▲ visual observation │ │
│ the agent perceives │ └────────────────────┘ │
└────────────────────────────────────────────────────────────────┘
"code controls consequences, renderer supplies the view"
The scene program runs in a simulator or game engine that maintains persistent world state and executes explicit interaction rules. After each action, the engine exports a structured condition (e.g., camera-view depth or surface normals) through a unified interface. A shared neural renderer combines that condition with an appearance reference, text, and visual history to synthesize the next observation. The agent sees it, chooses the next action, loop repeats.
This architecture turns a hard problem into two tractable ones: the part that must be exact (state, rules, collisions) is code; the part that must look real (vision) is a neural renderer. And because all environments export geometry through the same interface, they share a single renderer—so you can author new worlds entirely in code, without creating bespoke visual assets for every scene. Environment count and difficulty can therefore scale alongside the agents.
Part Two: Making Real-Time Interaction Feasible—Two Engineering Tricks
But “split rules and vision” is only conceptual. The engineering meat is in making the neural renderer fast enough and consistent enough to run an interactive loop above 30 fps. AgentGarten does this with two key moves.
The first move: adapt a pretrained video model to geometry conditions. Instead of training a renderer from scratch, AgentGarten takes a video world model and conditions it on the geometric structure exported by the engine. The renderer produces each new observation in fine-grained, short blocks so agents receive immediate visual feedback, while keeping visual consistency across extended rollouts (the world shouldn’t flicker from frame to frame). This is the “diffusion paints reality” half.
The second move: Adversarial Forcing, to make history prefilling differentiable through exact replay. Here’s the subtle training problem: a neural renderer must encode “what happened before” into how it renders the present. But if history is just replayed, losses on later predictions can’t flow back to improve how prior observations were encoded. AgentGarten’s Adversarial Forcing solves this by making history prefilling differentiable through exact replay—so gradients from later predictions update the encoder’s treatment of earlier observations. It then adds real-data adversarial supervision to improve visual quality beyond what pure self-distillation would give. The name “adversarial” signals a second critic that pushes the renderer toward outputs matching real visual distributions.
Why does this matter for agents? Because an agent that learns in a world needs that world to be both “faithful” (same action, same consequence) and “realistic” (observations follow real-world visual statistics). If the renderer drifts or flickers, the agent learns spurious correlations instead of stable rules. Adversarial Forcing is what keeps the world “honest.”
┌────────────────────────────────────────────────────────────────┐
│ Adversarial Forcing: keeping the renderer "honest" │
├────────────────────────────────────────────────────────────────┤
│ later prediction loss │
│ │ │
│ ▼ differentiable through EXACT REPLAY │
│ history prefilling ──────▶ updates how prior obs are encoded │
│ │ │
│ ▼ │
│ neural renderer ──── adversarial critic (real-data) ──────────▶│
│ │ pushes toward real visual dist │
│ ▼ │
│ next observation given to agent │
└────────────────────────────────────────────────────────────────┘
The result is a real-time (30fps+) interactive world where actions have reliable, visibly consistent consequences—the precondition for “learning by doing.”
Part Three: The Playbook Loop—Experience Inherited, Not Rediscovered
Having a real-time world is necessary but not sufficient for “fast evolution.” The other half is how agents reuse experience across generations. RL typically keeps an agent inside one policy and lets it stumble through millions of trials. AgentGarten chooses a different route: after each round, the agent reviews its attempts, records findings, notes questions—distilling an “experiment manual” (a playbook). The next generation inherits that playbook, refines it, and builds on it.
This is “Lamarckian” in flavor but not in genes: the playbook is not baked into weights; it is an explicit, readable artifact each new agent starts with. Concretely, in hide-and-seek the emergent strategies are striking:
- The hider learns on round 4 to drag the partition and build a shield.
- The seeker learns on round 10 to climb the wall with the ramp.
- After a failed jump, the seeker actively pushes the ramp closer, adjusts position, and tries again.
Those are not hard-coded behaviors; they emerged from the world + playbook mechanism. And the headline number is stark: agents learning from just 4 rounds, versus millions for a conventional reinforcement learning counterpart.
┌────────────────────────────────────────────────────────────────┐
│ the "playbook" evolution loop │
├────────────────────────────────────────────────────────────────┤
│ round n: agent acts in the real-time arena │
│ │ │
│ ▼ │
│ review (attempts / findings / questions) │
│ │ │
│ ▼ │
│ distill experience ──▶ write perceptual "playbook" │
│ │ │
│ ▼ │
│ round n+1: next generation inherits + refines the playbook │
│ → knowledge compounds across rounds, not rediscovered │
└────────────────────────────────────────────────────────────────┘
Let me make the “4 rounds vs millions” intuition concrete with a minimal simulation—the same demographic, but three training regimes, and you see why a “playbook” collapses sample complexity:
# why inheritance beat sample-heavy RL (illustrative, Python)
rounds_needed = {"vanilla RL (re-explore)": 2_000_000,
"RL + replay memory": 150_000,
"AgentGarten (playbook inheritance)": 4}
for regime, r in rounds_needed.items():
print(f"{regime:26s} -> ~{r:>7,} trial rounds to pass threshold")
# a tiny toy: each generation inherits 30% of prior knowledge as prior
base = 1000.0
for gen in range(1, 10):
pos = base * (1 + 0.5) ** (gen - 1) # compounding, not linear
if gen <= 5:
print(f"gen{gen}: inherited knowledge {int(pos)} (compounding)")
else:
break
The point isn’t the arithmetic but the mechanism: when experience is inherited as an explicit artifact instead of rediscovered inside one policy, learning efficiency stops being linear and becomes compounding. That’s the philosophical core of “evolving agents.”
Part Four: What This Means for Agent Training Infrastructure
AgentGarten is one paper, but it points at a broader shift in how the industry thinks about training agents. Three implications are worth holding on to.
First, “the environment” is becoming a first-class engineering surface, not a side detail. For a long time, agent research centered on models, rewards, and algorithms. AgentGarten reminds us that what agents can learn is bounded by the environments they practice in. A faithful-and-realistic world is itself the product—so building such worlds well (code worlds + shared renderer) becomes a competitive capability. Who makes “practice grounds” better will shape who gets smarter agents faster.
Second, “perception realism” and “state exactness” no longer have to trade off. The classic dilemma was “realistic worlds are fuzzily simulated, exact simulations look gray.” AgentGarten’s split—exact in code, realistic in neural rendering—breaks that trade-off. For practitioners, this is a reusable pattern: don’t force one component to be both exact and realistic; separate the two and connect them by a clean interface. The same principle extends beyond games to robotics sims, driving sims, embodied training.
Third, “explicit experience transfer” is a pragmatic alternative to sample-hungry RL. RL’s million-round sample cost is one of the biggest blockers to agent training at scale. AgentGarten’s playbook shows a complementary route: let agents write their findings down, pass them to the next generation, and treat knowledge as a compounding corpus rather than a secret locked in weights. It’s not a replacement for RL, but it suggests that the fastest path to “clever agents” may pair RL with explicit, inheritable experience—human-readable playbooks that engineers can even inspect and steer.
Part Five: Also Stay Sober—Where AgentGarten’s Boundaries Lie
The direction is exciting, but I also want to be clear about AgentGarten’s boundaries, lest it be misread as “AGI has arrived.” At least three limitations deserve sober treatment.
First, these worlds are currently high on “procedural” content and low on “openness.” AgentGarten is best at scenes with clear rules that code can depict (physics, geometry, games). But real-world complex tasks carry lots of vague semantics, social norms, long-tail common sense—hard to precisely model with a scene program. “Using code to build worlds” gives you precision, but also boxes you in by code’s expressive power. The worlds that can’t be written clearly in code are ones AgentGarten still can’t reach.
Second, rendered “realism” is visual realism, not necessarily full physical completeness. The shared neural renderer guarantees “looks like the real world,” but fidelity to real physics ultimately depends on how precisely the underlying engine models it. Realism ≠ exactness: rendering can fool the eye without guaranteeing it computes every mechanical detail correctly. For embodied training needing high-fidelity physics, this remains something to keep verifying.
Third, “playbook inheritance” solves learning efficiency but not safety and generalization. Fast compounding of experience is good—but if what gets passed down is “a shortcut of one particular world,” the next generation may collectively learn the shortcut askew. Experience inheritance brings speed, but also the compounding of bias and error. How to keep inherited experience correct and correctable is a harder problem than “learning fast.”
With these three boundaries clear, you can both believe AgentGarten opened the possibility of “letting agents evolve while playing,” and avoid expecting more from it than today’s capability allows. The progress is real, but the road is long.
Part Six: Four Most Practical Suggestions If You Want to Try It
If you’re genuinely moved by the “let agents evolve while playing” design and want to try it at small scale, here are four no-nonsense suggestions to help you avoid pits.
First, don’t chase realistic rendering right away. AgentGarten builds neural rendering into the loop because “visual fidelity brings behavioral generalization.” But for most hypotheses you want to verify first—“can an agent learn strategies in a small, rule-clear world?” “can adversarial environments force more robust behavior?"—a faithful-enough imaginary world may be good enough. Get the mechanism running first; pursue realism later.
Second, treat “adversarial forcing” as a default switch, not a nice-to-have. As covered earlier, the hidden wrong shortcuts in supervision signals are what make models “learn to do nothing.” AgentGarten’s experience is adding adversarial pressure inside the world so agents can’t slack off. When you build yourself, it’s the same: if the environment naturally lacks pressure to “push you to do better,” actively design some in. Training without pressure is often just training errors more skillfully.
Third, make good use of “playbook inheritance” to magnify efficiency, but be sure to give it an “exit to replay.” Letting experience compound quickly across new generations’ random initializations is AgentGarten’s speed lever. But remember the warning from Part Five—inheriting experience may also mean inheriting bias. So always give the agent the freedom to “break free of the previous generation’s experience and explore anew,” letting it put down shortcuts and change its way of life when it should.
Fourth, start from “being able to record every state seen during world browsing.” That memory-plus-playbook framework is essentially depositing “the process of seeing” into “transferable strategy.” When building your own, start small: first let the agent systematically record what it observes, then find ways to turn those records into instructions that guide next actions—if you nail these two steps, your “mini world” gains a locomotive for evolution.
None of these four is complicated, but each welds “making the world real” with “teaching the agent to learn.” The so-called evolution proving ground, in the end, is giving agents a soil that is both real and full of pressure, then letting them root down and grind out strategies by themselves.
Conclusion: When Agents Get a World They Can Grow In
AgentGarten, on the surface, is “an arena for agents to play in.” But its real weight is architectural and philosophical: it demonstrates that “agents that keep evolving through interactive experience” can be engineered, not just imagined. By splitting exact rules into code and realistic vision into a shared neural renderer, by making history differentiable through Adversarial Forcing, and by letting each generation inherit a playbook, it turns agent development from “train one policy until it memorizes” into “raise a lineage that keeps learning from a world that scales with them.”
And for anyone who wrestles with agents at work, the practical lesson is the encouraging one: you don’t need billions of trials to teach an agent to do something new—you need a world where actions have reliable, visible consequences, and a mechanism to carry experience forward. Build those two, and learning follows almost naturally. That is the quiet revolution AgentGarten is pointing at: the day agents get a good world and a way to remember, “evolving while playing” stops being a metaphor.