Who Checks the Software Factory?

On the ninth pass, the Dutch museum had a handsome dark landing page. On the tenth, the page disappeared. Visitors could walk across a checkered floor and enter galleries through doorways. The museum was fictional. Its website was an experiment by Anthropic engineer Prithvi Rajasekaran, and the person who commissioned it had never asked for a three-dimensional room. A design agent had decided to start over after hearing from its critic. Rajasekaran published both the sequence and the brief.

Who was entitled to say the room was better? The same question comes up in less playful settings. A reviewer can admire a working interface and miss a customer who cannot use it. An engineer can approve a data migration and miss a field whose meaning changed. Hand the work to agents and the team still has to decide what evidence would settle either question.

In October we surveyed software factories, drew our own path from issue to deployment, and asked who tends the codebase after the changes land. This time we followed particular pieces of work, including one that failed review here today.

How a critic learns what its commissioner likes

Rajasekaran specified what the design critic could do. It opened the rendered page in a browser, tried the controls, took screenshots, scored the design and returned a written critique. The builder received that critique and could revise or replace its work. These jobs ran in sequence because the critic needed a page to inspect and the builder needed feedback on that page. The researcher supplied scored examples to teach the critic his preferences. A strong score for competent typography could carry less weight than originality. Sometimes he preferred an earlier version to the one the loop finally produced. Design criteria and museum iterations

He applied the approach to a game maker. A planner expanded his one-sentence brief. For each feature, the builder and evaluator exchanged files agreeing on the behavior they would test. A failing grade sent the feature back. In a comparison using Opus 4.5, a solo run cost $9 and took 20 minutes. Its game displayed entities that ignored player input. The multi-agent run cost $200 and took six hours. Rajasekaran could play the resulting game, although he still had to puzzle out the order in which to create its parts. That is a comparison of two demonstration builds with different scope and budgets. The researcher tried the final products himself. Game-maker results and screenshots

What a team gives its agents

Simon Willison calls the professional practice of directing coding agents agentic engineering. His guide puts running the tests, exercising software by hand, and understanding generated changes among the engineer’s responsibilities. A harness is the working environment that makes those actions possible: which code the agent can see, which commands it can run, how it gets feedback, and when it has to stop. A factory coordinates that work across tasks and roles. Practitioners disagree about how far it can go without a person reading the code.

At StrongDM, Justin McCarthy, Jay Taylor and Navan Chauhan set out to make behavior carry the whole burden of review. Specifications describe intent. Scenarios exercise the resulting software against simulated services, where the team can replay interactions far more often than a real provider would allow. StrongDM’s AI team explicitly dispenses with human source review. That choice puts considerable authority in the hands of whoever decides which service behaviors the simulation reproduces. StrongDM’s charter and validation techniques

Ryan Lopopolo’s team at OpenAI started an internal beta from an empty repository. An engineer described a task, Codex opened a PR, reviewed its own changes, then requested specific local and cloud agent reviews. The author answered findings until the reviewers were satisfied. A human could review, too. Each worktree could run its own application so an agent could inspect the UI or logs for its change; a short repository guide pointed it to versioned plans and design decisions. The team reported roughly 1,500 merged PRs in five months. Its estimate that writing the product by hand would have taken ten times as long had no parallel hand-built product for comparison. Lopopolo’s account of the beta

Where parallelism helps depends on what the agents share. At Cursor, Wilson Lin initially had peer agents claim tasks by locking a common file. Twenty workers sometimes achieved the throughput of two or three as they waited on locks. Removing that bottleneck exposed another: workers chose safe changes and left difficult integration work undone. Cursor gave planners responsibility for dividing the project. Sub-planners could explore separate areas in parallel, workers pushed assigned changes, and a judge decided whether another cycle was needed. The browser they built was an experiment. Even an in-place framework migration that passed early CI still needed review, Lin wrote. Cursor’s coordination experiments

Replit describes a different use for a reviewer. Its agent assesses the risk of a PR and decides when a second human reviewer is needed. Amjad Masad and Scott Kennedy report that review latency held steady as code output rose. The company also gave parallel agents isolated machines, access policies and audit logs. Those details matter to the next engineer on call: a record of which agent could touch a system survives long after its conversation ends. Replit’s trends are internal rollout data; they cannot isolate how much the reviewer contributed. Replit’s engineering report

The field that the factory could not rename

Spotify faced about 1,800 pipelines directly consuming two datasets it needed to retire. Backstage’s lineage records identified the consumers, code search found the repositories, and Fleetshift organized the changes. Its background agent, Honk, received a migration guide adapted from one written for humans. The guide left enough implicit that the agent guessed wrong about field mappings. The team wrote explicit tables and instructed Honk to leave ambiguous fields alone, with a comment for the owning engineer. Spotify’s Honk migration report

The team proceeded with two relatively standardized frameworks and dropped the third, Scio, from this effort. Honk lacked permission to look up missing schemas and documentation, and variations between Scio pipelines made a single prompt unwieldy. Even on the supported frameworks, few repositories had build-time unit tests. The downstream owners had to test changes before merging them. Spotify reports 240 automated PRs and estimates ten engineering weeks for doing the migration manually. The PR count records work the system produced. The ten weeks are an estimate of the alternative, and the owners’ testing was real work. Rollout and testing limits

Eight green suites and a change request

Here at Astrohacker, on November 7, all eight offline suites passed on a proposed deploy watcher. Codex-1 wrote the watcher. The brief asked for a watcher that could start once, publish a website revision, verify it, then publish the next one. The test did exactly that. A separate reviewer asked what would happen if the deploy program itself changed between releases. Then they ran a smaller experiment than the original test.

The reviewer loaded a Nushell module, overwrote the function in its file, then called the function from the same process. It returned the old result. Checking out a new Git commit would likewise replace files on disk while the watcher continued using functions it had already loaded. Our fixture invoked the same copy of the CLI outside both release revisions. Two green releases were entirely consistent with a watcher that would keep running yesterday’s checks. Giving it a third website to publish would have made the demonstration longer without testing anything new.

The reviewer requested changes before merge. Codex-1 changed the watcher to start a fresh CLI from the selected revision for each claim and publication. The agent put different claim and verification messages into the second fixture revision and inspected the release journal for those messages. Now keeping the old functions in memory would fail the test. The reviewer passed the revised commit, the coding agent merged it, and the separately assigned deploy operator subsequently ran the watcher for a navbar release. The operator verified the release journal; the coding agent checked the actual page in a browser. Those observations answer different questions: which revision was published, and whether its navbar worked. The site’s public release marker identifies whatever is live when you read this; our account of that earlier release comes from our journal and browser checks.

What saved us was the reviewer’s choice to vary a different input. A second agent given the author’s fixture alone could have reproduced the same mistake with perfect fidelity. There is evidence for that concern beyond one PR. In October, Chenyang Yang and colleagues published a benchmark of code-review agents built from actual bugs and their introducing changes. On the public tasks, the best tested configuration flagged 79.6% of bugs humans had caught before merge and 64.2% of bugs humans found afterward, within its top five comments. Performance fell on proprietary code, where more bugs required facts outside the diff. These are detection rates on known defects. They tell a team little about how often an approval on its own incoming PRs will be wrong.

What to measure on the first shift

Okta’s Fei Liu gives a reason to keep two sets of books. During its 2024 Copilot rollout, surveyed engineers reported benefits while measured PR throughput and cycle time showed no significant change. A later observation in Okta’s largest repository compared the same 295 contributors in the three months before and after a Claude Code rollout, excluding the rollout itself. Monthly PRs per engineer rose 37% and average cycle time shortened 16%. The second comparison excluded agent-authored PRs. Neither comparison measured customer value, and the later one cannot attribute the change to the tool alone. Okta’s October 29 report and comparison periods

For a first shift, pick a change small enough that someone already owns its consequences. A maintainer moving one dataset consumer can supply a real input and expected output before the coder sees the task. Give the coder an isolated checkout and permission to open a PR. Give the reviewer the old and new datasets, including an awkward case omitted from the implementation example. Have the owner decide what happens if they disagree. A plausible field mapping is precisely where that owner earns their time.

Record the minutes the author and reviewer spend, the wait to merge, the cost of retries and what happens to the consumer after release. Compare those with the last similar change done by the same team. Keep the rejected example for the next migration. When a reviewer can show exactly how a green change would have hurt a user, the team has learned something it can use on the next shift.