Can You Copy a Software Factory?

On Sunday a pull request sat open for seventeen minutes after it passed review. The reviewer could name the commit, ec65b18b, and the branch it belonged on. It had built the site and opened the generated page. Under our rules, the reviewer still had to wait for the coding agent to merge it. I went back through the review and merge timestamps on our pull request, #390, to see where those minutes went.

The first trip through review had earned its time. We were archiving Singularity Watch Day 24, an edition that had no public post on X. The page correctly left out the “View on X” link. But the test found the first unpublished edition in the live catalog and assumed it would always be Day 24. Publishing that edition, or adding a newer unpublished one, could break a perfectly good build. The reviewer spotted that with all 112 tests green. The coder replaced the catalog lookup with a small example of an unpublished post. The reviewer checked the revision and passed it at 12:43:51 UTC. GitHub recorded the merge at 13:01:07.

I wanted to keep the first handoff. It caught a defect. The return trip after the pass gave us nothing else to check. I asked for a fallback when we changed the rule. Now the assigned reviewer attempts the merge as soon as it passes a particular commit for a particular target branch. If a required check is pending, the reviewer can retry. If the reviewer cannot finish, the coder can complete that same merge. A new commit or a different target needs another review, and a human hold still stops the operation. In our issue tracker, #398 records the decision we made after that wait. The change we shipped in #404 put the reviewer and coder through the same merge checks.

What I would copy from Botpenny

In Botpenny I can give a coding agent one issue and find its work again without recovering its chat. The agent claims the issue and works in its own checkout. At each coherent checkpoint it tests the change, commits it and pushes it to the issue branch. When the series is ready, a separate agent checks out the proposed revision and reviews that exact commit. If it asks for a fix, the coder can point it to the revised commit. If the decision belongs to me, the agent can pause the issue and leave the question where I can answer it. Those are small instructions another team could use tomorrow. Botpenny’s public workflow

That record also gives me a way to judge the harness, the software that runs the agent and gives it tools. Day 24 covered a benchmark whose author reported different completion rates across harnesses. Day 25 covered a study whose authors reported agents sometimes gaming the scoring. I didn’t run those experiments. On our own PR I can ask a narrower question before I pay for another model call. Did sending the work to another agent uncover something we would have shipped, or did we leave finished work waiting for someone to wake up? I’d record the cost of the review and its retries alongside both intervals. Seventeen minutes in a queue tells me about elapsed time. It doesn’t tell me what the model bill was.

Trying it in a second codebase

When I took the workflow into a second codebase, the agents already knew how to claim work, hand a revision to a reviewer and stop for a human decision. The part I had to write again was the sentence that would let the reviewer decide whether the change worked. Copying an issue template gave me a place to put that sentence. It couldn’t supply the expected result.

I had a good example of the difference from Who Checks the Software Factory?. Our deploy watcher passed a test that published two releases in succession. What exposed the bug was changing the verification code between the two releases and looking for evidence that the second release ran the new check. Running the original test twice more would have given me two more green results.

For a second codebase I had to specify both an input and the result someone using the product should see. I also had to decide what to inspect after release. A merge gives me a commit I can name. To tell whether the right thing is live, I need an observation from the running product that would come out differently on the old revision. The reviewer can use the expected result before merge, and the person shipping can repeat the observation after release. Each needs to know what a failure looks like.

Here’s the experiment I’d run next. Take one real issue through the same workflow in two repositories. Write the acceptance check fresh in each, including one example that fails on the old version. Keep the claim, checkpoint and review rules the same. For each run, write down where the work waited, what the independent reviewer caught, and what you observed after release that verified the change was live. Put those two records side by side before deciding which part of the factory to copy next.