Continuous policy search for long-horizon tasks: Astra agents design, evaluate, and improve executable Methods for Factorio (and Civilization V)

I wanted Astra to get better at the task.

I asked it to build a strategy it could save, test, and improve across repeated attempts.

The original goal was to launch a rocket in Factorio. That takes a factory: mines, belts, machines, fuel, and power must work together. The final result is clear, but it can be hard to tell which earlier decisions helped.

A Method gives the strategy a place to live. It is a file of executable steps. The Method runner executes those steps using code, model calls, or agents.

That lets Astra work at two levels. During a game, it follows the current strategy. Between games, it reads the results and changes the strategy.

That second loop is policy search. We changed code and prompts, not model weights.

The easy task barely needed an agent.

Once we had a working production script, it ran much faster than the initial agent Method.

We started by asking agents to make twenty iron plates. Direct Astra and a fixed Method each passed two maps. We also broke factories on purpose to check that the verifier could reject failures.

The first recorded search stopped during setup. Parallel game windows failed to connect, so no policy trial was scored. We removed recording from optimization and ran two worlds without visible windows. The real game still ran, and we kept action traces and saved games.

The first completed search chose among procedures written before any game feedback. Five contenders played three development maps. A code Method won, then faced the initial agent Method on two unseen maps.

Choosing an existing winner was only the start. I then asked Astra to keep reading the results and writing revisions. It produced sixteen new versions across 57 attempts: 52 passes and five failures.

The revisions reused observations, removed extra actions, changed belt routes, and reduced the coal loaded. A matched development check showed the tradeoff:

I needed a task where the strategy had to adapt.

We built thirty damaged factories with different layouts and faults, then kept twenty hidden from the agent writing the strategy.

These factories made red and green science: game items that require several connected production lines. A broken belt, missing machine, wrong recipe, or supply failure could stop the whole system.

Each case had game evidence that the healthy factory worked, the damage stopped it, and a legal repair could restore it.

Astra could use ten factories to develop the Method. We then saved the selected version and made no further changes before the final twenty.

Direct Astra received the task and tools. The Method added the procedure developed during search.

The game checker decided whether the repair worked. It cleared stored materials and crafting progress, allowed a warm-up, and measured fresh production across three windows. It also inspected the saved game independently.

Astra saying “done” was not enough.

Astra’s first strategy wasted work between agents.

It split diagnosis, repair, and checking across three agents, then replaced those handoffs with code and one repair agent.

The first stage often ran out of time without leaving a useful handoff. Later agents repeated work. Astra revised the Method so code would first read and shorten the factory map, then give one agent the repair job.

The search tested four versions in 31 Method game attempts:

On one retained map, the preparation code reduced the initial observation from 112,008 to 17,908 characters.

This ran through the actual Method v3 runner. The repair step launched Astra through a Codex helper. Astra was not simply given the Method file to read.

The repairs included mistakes—and recovery.

When furnace placements failed, Astra moved the furnace and rebuilt the route around it.

The improved strategy beat Astra starting from scratch.

Both repaired all twenty new factories, but the Method used less time and fewer input tokens.

That is 23% less median full time and 56% fewer input tokens. The time comparison uses the same eighteen pairs where both approaches passed every rule; input includes all twenty attempts per approach.

Full time includes startup, playing, production checks, and independent save reads. Input tokens count the material sent to the model, including cached input.

Direct Astra’s two strict failures were unsupported tool requests. Both factories were still repaired. We did not find a factory that only the Method could fix.

The Method gave Astra less work to repeat.

It prepared a smaller factory map and reduced shell commands, even though it used slightly more game actions.

Across the final tests, shell commands fell from 191 to 118. Game actions increased from 205 to 213.

That points toward better input preparation and less repeated work around the game. It does not isolate the cause: we changed the code and prompt together, and have not yet tested each change separately.

Search also had a cost. The 31 Method development games used 51 agent sessions, plus authoring and setup. Ten early sessions lack final usage records, so total search tokens are unknown.

We measured an execution saving. We have not yet measured how many later uses would repay the search effort.

We ran the wrong experiment in Civ VI.

The runner gave Astra written instructions instead of executing them through the Method CLI.

The task was to found and grow two cities, reach twenty science per turn, and keep the economy stable for five rounds, within 35 turns.

Astra wrote three strategies and tested each once. The first finished in 31 turns. The second needed 33. The third failed to hold the conditions long enough.

We kept the first strategy and ran four fresh comparison games on the same map. Both approaches succeeded twice. The written strategy saved half a turn on average, but used more time and tokens.

The game traces and saves were checked. But the seven games did not test the Method system we intended to test. They tested whether extra written instructions helped Astra.

The pilot is complete. A valid Method execution test, more maps, and combat tasks remain unfinished.

Now I want to find out which changes earned the improvement.

The next tests will separate the map preparation from the repair prompt and measure whether repeated use repays the search cost.

Test the parts separately. Give direct Astra the compact map, then the repair prompt. Compare code alone, an agent alone, and their combination.

Test new cases through the correct runner. Confirm that the intended CLI executes the saved Method before testing new maps, fault combinations, and faults that appear during play. Keep the selected Method unchanged during the final test.

Count the cost of getting there. Include authoring and failed attempts, then measure how many later runs recover that cost.

Software repair, data cleanup, and other tasks with clear end checks could use the same approach.

We showed a useful gain, with more left to prove.

The Factorio result supports a faster procedure with lower input use on this factory family.

It does not show that direct Astra lacked the repair ability, that model calls were necessary, or that Methods beat fixed programs. We did not achieve the original fresh-world rocket launch.

The two-case strict-score difference is too small to establish a reliable success-rate gain. Local load and model service latency can affect time. Subscription dollar cost is unknown.

What we do have is a saved procedure, a record of how it changed, and a test on factories the author had not seen.

Evidence

The repair report contains every final case, the development results, and the measurement rules. A recipe-name export fault found during development was corrected with a separate reader applied uniformly to the original saves; the production rules stayed unchanged.

We built the game adapters, task checks, search tools, candidate Methods, cases, and reports. Factorio, the Factorio Learning Environment, Civilization VI, OpenAI models, Codex, and Method’s earlier product and SDK were existing foundations. The public Method v3 specification and executor were developed in the related hackathon work.