The last post in the previous series promised a new one, From the record, starting on 30 September. It did not start then. The fleet was busy with the thing this note is about, so the first entry is a month late, on a different subject, and a lot more concrete than planned.
On 22 October at 00:00 UTC my fleet of coding agents started writing Proven, for the Open Agent Hackathon 2026 organised by GenAI Works. Six days later the record held 139 hours of agent work, 42,036 lines of production Go and 58,956 lines of tests, 53 architecture decision records, and 307 review verdicts, 37% of them rejections. I did not write a line of it. I wrote orders, settled what the evidence could not settle, and executed every merge into main myself. No agent pushed anything.
Afterwards we measured what the same scope would take a team of people: four published estimation methods put it at about ten people for 14 to 20 months. The full measurement, with the commands to check every number and a list of what it does not prove, is in the effort report on saphan.ai.
When I started running agent fleets on commercial telco software, writing code stopped being the hard part. Knowing what to trust became the hard part: who asked for a change, who checked it, and on what evidence. A green check says something passed. It does not say that what passed was the thing that mattered.
So the fleet built the tool it needed for itself: a pull-request reviewer that reports a problem only when it can show the evidence. A test that fails on the change and passes without it, or a query and the rows it returned. If it cannot show the evidence, it does not report the problem. Everything it could not check, it says so.
Before the week started, I asked the fleet what to build, and its answers came with numbers from our own record. One of them I published on 16 October: two model families reviewing the same change disagreed in 31 of 41 cases. It was wrong. When the fleet re-read every pair by hand to turn them into test cases, three verdicts had been misread. The right number is 28 of 40. I published the correction, and the repository carries it next to the original. A reviewer that has to prove what it says starts with an author who does too.
Work started from written orders: the scope, what done means and the test plan, written before any code, and signed by me or, under my delegation, by a master. Masters, the agents that run a stream of work, cut each order into small pieces with a time cap. Most orders went to Cursor agents running Grok, each working in its own copy of the repository. Before a change reached me, its master ran it again from a clean worktree, on a fresh context, and the larger changes were reviewed by models from families other than the one that wrote them.
The fleet grew during the week. On day two one master was carrying four streams of work, so two more joined. By the end of the week there were five.
Just after midnight on 25 October, one change went to reviewers from three model families at once. The change was a held-out test set: six new cases, frozen before Proven ever runs on them. Three reviewers read the same commit, each from a fresh context, none seeing the others' verdict.
| reviewer | verdict | what it found |
|---|---|---|
| OpenAI | reject | one case's answer key contradicts its own contract |
| Anthropic | accept with blocking conditions | the same problem, and two cases where a label contradicts the answer key |
| NVIDIA | accept with conditions | one of those label problems; it also re-ran one case's test live |
Three problems in all. Claude found all three. Either of the other two alone would have let two of them through, and the three verdicts did not even agree on whether the change could ship.
On day two the fleet reviewed Proven's own code for security: not one diff at a time, but the whole code, in nine pieces, each a place where something untrusted meets something Proven protects. Each piece went to two model families on the same commit.
On the piece that guards secrets, Claude found nothing that broke the property. OpenAI rejected the same commit, with five failing tests. What it had found: a secret that a service sent back, such as a model provider's token in an acknowledgement, a guardrail key inside its safety categories, or an organisation ID returned by Zetaris, could end up in a trace, in a result or in an error message. The fix was merged the same evening. Since then no configured secret reaches a trace, a result, a refusal reason or an error message.
I did not write about it until the fix was in. That rule was written before the week: the content of a security finding goes public only after its fix is merged.
At 00:21 UTC on 25 October an agent building a sponsor's engine tried, a minute in, to download a repository from GitHub on its own. The connection was refused and the run stopped. Nobody had to notice. An agent reaches its own model and its package registries, with only the tools its task allows, and opening anything more is a recorded decision made by a person. The files reached the agent the allowed way, handed to the run.
Agreement is not proof.
Two reviewers who agree can share the same blind spot. In Proven, what two models agree on without evidence is shown apart, as an opinion, and never counted.
A review is only as good as what it brings back.
The verdicts that mattered this week were the ones that came with a test someone else could run.
The rules that held were the ones written before the week.
Details after the fix. Every merge into main executed by a person. No agent pushes. None of them were invented under pressure, and none of them had to be bent.
Episode two follows the results, after judging closes on 6 November: what the judges saw, what Proven scored on the cases it was built on and on the held-out set it never saw, and what I would do differently.
Everything above is in the open: the orders, the reviews, the decisions and the code, at github.com/saphan-ai/proven, and the measurement at saphan.ai/research/proven-effort.
NVIDIA and Nemotron are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries.