Agents will tell you they tested it

The thing every agent I’ve used does

In my experience, ask Codex or Claude to fix a bug in a user-visible flow and it will fix the bug, run the unit tests, and report that it tested the change. What it usually means is that the tests passed. Sometimes it means it read the code and is confident. It very rarely means it started the app, logged in, went to the screen, and did the thing a user would do.

That gap caught me out enough times in the first half of 2026 that I stopped trusting the word "tested" in a handoff. The fix I wanted was not a better prompt. It was a place where the agent could actually do the testing, and a set of words it had to use afterwards that couldn’t be fudged.

What it is

To be clear about scope: this is my own development fixture, running on my Mac, that my agents share. Colleagues run the platform locally their own way. It isn’t a team service and I don’t claim it as one.

One checkout, kept clean, reset every morning to main. An agent that wants it claims a lease with its own name, says exactly which commit it is validating, and gets the checkout for as long as it keeps a heartbeat going. When it’s done it has to clean up its own processes, databases and containers and release the lease. The next agent won’t get the slot until the previous one has left it the way it found it.

That’s most of it. In October it became two slots with a queue, because two agents kept wanting it at the same time, but the idea didn’t change.

The words that matter

The part I’d actually defend is the handoff. Every validation that uses the runner has to end with four lines:

golden_runner_journey: PASS | FAIL | BLOCKED
requested_ui_path_exercised: true | false
independent_reproduction: PASS | FAIL | NOT_RUN
request_satisfied: true | false

The point of splitting them is that they stop an agent trading one for another. A passing database check does not upgrade a blocked UI journey. A UI run on a slightly different commit doesn’t count as a run on this one. If the app couldn’t be started or logged into, the answer is BLOCKED, written down as blocked, with the boundary it hit. It is not "tested" with a caveat buried in paragraph four.

Since August a PASS on the exact commit has been the gate before any PR that touches something a user can see. The order is implement, commit, validate that SHA, pass, then open the PR. Not the other way round.

What I learned from it being annoying

The runner was a nuisance for the first month. Agents would claim it and then fail to start the app, or start the app and fail to log in, or log in to the wrong database, and each time they’d report the failure as if the code was at fault. When I went through the failures they were nearly all the same shape. The agent had crossed four separate boundaries without noticing any of them: did I own the runner, is this the exact source I think it is, are the services actually up, am I logged in with data that exists. A green checkout is only the first of those.

So the runbook grew a preflight that checks each boundary on its own and names which one failed. That cut the noise down a lot. It also produced a rule I now apply to everything agents do, which is that a setup failure and a product failure are different results and must never be reported with the same word.

The honest version

I don’t think the Golden Runner is a product, and I’m wary of over-engineering it. It’s a checkout, a lease file, a few Ruby scripts and a page of rules. What I’d keep from it, if I had to throw the rest away, is the handoff. Four lines, no synonyms, and an agent that has to say BLOCKED out loud.