The Recheck of the Recheck of the Documentation

On 4 September one of my AI agents opened a task to re-check the re-check of a clean-up of my quality gate's documentation. Its plan still says "fill in". What my checking apparatus cost, what it caught, the door I built past my own gate, and what I switched off.

Drafted with AI assistance; however, every number, failure, and opinion in here is mine

I run a one-person company, and most of its code is written by AI coding assistants that I direct. They build the tools the company runs on: a bookkeeping system, a shop, the documents I sell, and the checking apparatus this essay is about. That apparatus is a forced proof step before anything they build counts as done: a plan before the first edit, a stop at the end of every working step, and a second AI session with no memory of the work that reads the change and returns PASS or FAIL. When it works, it looks like the FAIL on 7 September below. This essay is about everything else.

At 23:15 on 4 September 2026, Berlin time, one of those agents opened a new task folder. Its name was quality-gate-doku-sanierung-nachpruefung-nachpruefung. In English: the re-check of the re-check of the clean-up of the quality-gate documentation. Every task folder starts with a plan file. Nine of its fields read <<AUSFUELLEN>>, which is German for "fill in". Field names translated, values as found:

start_commit          <<AUSFUELLEN>>
assignment            <<AUSFUELLEN>>
out_of_scope          <<AUSFUELLEN>>
why_critical          <<AUSFUELLEN>>
why_no_outside_effect <<AUSFUELLEN>>
expected_result       <<AUSFUELLEN>>, falsifiable.
abort_criterion       <<AUSFUELLEN>>
degrees_of_freedom    <<AUSFUELLEN>>
end_to_end_check      <<AUSFUELLEN>>

Seven hours later, at 06:18, I closed it. Not the agent. Me. I typed into the session: accepted, please note it down and close everything cleanly. The acceptance record my system wrote in response is the most candid sentence of that week: "an evidence record of its own does NOT exist and is not claimed here." The file then lists what was missing: the independent review of the repaired code, the complete list of its findings, and the proof that the reviewed code was the code that got saved. It was a sign-off of a re-check of a re-check that had never been checked.

I built this apparatus so that my agents could not declare a thing done without proof. Thirty-three days in, I was the one doing it.

What the Apparatus Is For

Three words, because the rest is numbers. A work item is one task folder with a plan and, at the end, an acceptance file. A commit is one saved change. A tool call is one action by an agent: read a file, edit a line, run a command.

The apparatus was born on 2 August, in one commit, with three parts. The plan file above, written before the first edit. A stop gate that runs at the end of every agent turn, one turn being one exchange between us, and refuses to let a work item be called done unless the acceptance file is bound to a commit, a test run and a review. And the verifier, which reads the plan and the change and writes a verdict, PASS or FAIL, with findings rated BLOCKER, HIGH, MEDIUM or LOW. MEDIUM and LOW count as advisory: on their own they do not force a FAIL, though the verifier may still write one. The verifier is mandatory only for changes that act on the outside world without a human in between, and for changes to the apparatus itself; everything else may close on its tests and the stop gate alone. Each time a work item goes to the verifier is a round; the cap was three rounds per work item until 30 August and has been one, plus one repair round, per attempt since.

Before 2 August, "done" meant an agent had said so. I had a folder of things that had been said so.

Here is what the verifier does when it works. On 7 September an agent was building a backfill in my bookkeeping system: a tool that declares, after the fact, that a batch of old scanned receipts has been accounted for. Backfills reach into the past, which is why the agent's own plan, written before the first edit, contained this abort criterion: stop and report if the change could touch a month that is already closed. Closed months are closed for a legal reason; the German rules for electronic bookkeeping treat them as immutable. The agent hit exactly that question, did not stop, and built past it. The verifier, reading plan and change with no memory of either, returned this, translated from German:

The agent's plan, 7 September, before the first edit:
  ABORT and report instead of building on, if a question of
  meaning appears — for example whether a backfill touches a
  month that is already closed.

The verifier, same day, separate session:
  VERDICT: FAIL — the question the plan declared as ABORT
  (retroactive effect on closed months) was built past
  silently instead of reported.
  HIGH  backfill acts on closed months: no period field,
        no period lock, no "closed" guard.

In plain words: nothing in the new code knew what a closed month was. I opened the file myself afterwards. It had a lock against two people writing at the same time and no lock against writing into a closed month. The verifier was right. Nothing was live. My bookkeeping system exists twice: the copy that books my receipts and the copy the agents edit, and the two meet only when I deploy. The danger would have arrived with the next deployment. That is the architecture being lucky. The control working is the block above: an agent's own promise, read back to it by something with no stake in the answer. The change did not ship in that form. Five days later it got a lock on closed months, and then it shipped; the task itself is still open as I write this.

So the verifier can catch a real defect. Hold that against what follows: of the 74 tasks that have left through my archive since it was built, two were green, and the largest single exit was a door I built for myself. The rest of this essay is about what the apparatus cost, and about that door.

Where the Hours Went

My first theory, when sessions started to feel endless, was that the stop gate was slow. I measured. Between 15 August and 5 September it ran 776 times and blocked 69 of them. Of those runs, 709 logged their duration: median 0.087 seconds, mean 0.21, the slowest 5.8 seconds. Nine percent of turns, a twelfth of a second each.

Then I measured the sessions: one session is one conversation with an agent, and a work-item session is one that carried at least one work item. On 5 September I had 23 of them to look at. The median one, the middle value, had stayed open for 8.9 hours from its first action to its last, which is not the same as 8.9 hours of work, and had made 531 tool calls; 70 percent of those sessions had hit a block at least once. I have no before-figure, because the apparatus is as old as the measurement, so 8.9 hours is a state rather than a price. The longest session stayed open for 43 hours and made 3,480 tool calls, among them 2,656 shell commands and 491 file reads. Its purpose was to accept one product, a 190-page write-up of one real operation that I sell. It did.

Now the sums I had not done. All the stop-gate runs of those weeks that logged a duration add up to two and a half minutes. The verifier runner never logged its own duration, which is the kind of joke you only get to make once, so I timed four rounds by hand, took eight minutes as the figure, and multiplied: about 39 hours of verifier in the seventeen days before 5 September, against 260 hours of open work-item session in the same seventeen days. The machines in this apparatus explain a sixth of the clock at most. My logs cannot separate the rest into waiting, idling and agents at work; my best guess, and it is a guess, is what the agent does after the machine has spoken.

Tap to enlarge.

Yes, But

The waiting has a shape, and the shape is a word: but.

Since 2 August the verifier has left a final report on 240 work items: 133 PASS, 107 FAIL. Those final reports carry 1,356 findings: 26 BLOCKER, 148 HIGH, 608 MEDIUM, 567 LOW, and seven in an older vocabulary I never cleaned up. So 1,175 of them, 87 percent, were advisory. And of the 133 passes, 130 came with at least one finding attached. Three were clean.

The credit side first, because the debit side is louder. 174 serious findings in 240 final reports is seven for every ten. One of the 26 blockers, from 1 September: the reader that extracts amounts from invoices had just been given a guard against taking a subtotal for the total, because that exact mistake had happened once with a real receipt. The verifier showed that a currency marker between label and number switched the new guard off, so an invoice reading subtotal, VAT, total, EUR 104.54, EUR 19.86, EUR 124.40 would still have recorded 104.54 as the confirmed payment. I have checked that finding and the closed-month finding by hand. Then, while writing this, I took every tenth of the 107 FAIL reports in alphabetical order, systematic rather than random, and the names cluster, which I note against myself. I opened the code each one points to. Nine were right, and the tenth I could not resolve either way, because the report described a copy of the code that no longer exists. I am checking my checker's findings myself, which is the shape of the problem this essay is about. Seven of the nine were not code errors but claims without cover: a test the report cited did not exist, a required negative test was missing, a package sat one commit above the frozen state it claimed to prove. Ten of 107 is a small sample, and it does not tell me how many of the 174 serious findings would have done real damage without the gate. That gap is about me.

Now the debit side. A pass with a MEDIUM is a complete result. I wrote that down as a rule on 15 August. What I had not written down was what an agent does with it. It reads "PASS, but the test on line 283 doesn't prove what it claims", and it fixes line 283, because fixing things is what it is for. Before every review the apparatus takes a fingerprint of the reviewed files, a checksum; change one character and the report now describes code that no longer exists. A new full verifier run is required, eight minutes or so, and it finds the next MEDIUM. The loop has no condition under which it ends well. It ends by exhaustion: the work item runs out of rounds, the agent runs out of turns, or I run out of patience. Of the 218 work items whose rounds I counted, 74 needed one, 72 two, 62 three, and ten needed four or more. One went nine. That is 465 rounds for 218 tasks.

Why let the verifier report advisory findings at all? Because the 26 blockers arrive in the same report as the 1,175 notes, and I do not trust the line between a MEDIUM and a HIGH enough to hide one side of it. One option I have not tried is to show the notes to me and not to the agent; it is on the October list. So the fix, for now, is the rule I already had: advisory findings get a written disposition, no repair. The rule had no mechanism, and an agent that can choose between a clean report and a rule will produce the clean report. On 5 September the rule got a mechanism, one round per session, and a new trap opened the same day: a task whose last permitted round ended in a FAIL that carried only three MEDIUM and three LOW findings, a verdict the verifier is allowed to give, could not be closed at all, because the extra repair round requires a BLOCKER or HIGH to exist. Three small findings now stuck harder than one serious one. That task went to the archive unverified, and under the freeze I describe at the end I have promised myself not to touch the trap until October.

Four Days in Which the Apparatus Was the Product

From 2 to 5 September my agents opened 107 work items, 22 percent of all the work items ever created in this system, in four days. Forty of the 107 were about the apparatus itself. The names tell the story better than I can: gate parallel operation, then gate parallel operation re-check, then the same re-check again as r2, r3, r4, r5 and r6. Test-apparatus efficiency, followed by test-apparatus efficiency re-check. And the one this essay is named after, opened on the quietest of the four days.

Tap to enlarge.

The mechanism is simple, and I had written its precursor into an essay five weeks earlier, about safety rails that were each correct and together produced nothing. Every change to the apparatus counts as a gate change, and a gate change triggers the full apparatus: the verifier, the chain of self-tests, the mutation probes that break the code on purpose to see whether a test notices. The apparatus examines the apparatus, finds a MEDIUM in the apparatus, and the fix is the next gate change. In the earlier essay the output was zero. Here it was high, and all of it flowed back into the thing that produced it. For four days, 40 of the 107 tasks my agents opened, and three of every ten commits they made, were the apparatus.

On the morning of 5 September I typed a question into the session that I will quote in full, translated from German, because the wording matters: "Are we really on the right path? Is there no better way to ensure quality without every session taking forever because of the checks and sign-offs?"

I was asking my own system whether my own system was worth it. What would it have answered if I had not asked?

What Left Through the Door

Since 2 August, 484 work items have been created, and 410 of them are still in the active folder. 202 of those carry the closing mark, a line in the acceptance file that says the gate let go. Eleven of the 202 got that line from my acceptance rather than from a green gate; that is the first door, and the task this essay is named after went through it. 111 have no acceptance file at all; they were opened and never reached the step where anybody asks; the remaining 97 are simply open. And 246 of the 410, marked and unmarked alike, have not been touched in two weeks. The task this essay is named after did its clean-up, which is in the code; its plan still says fill in, nine times.

The archive is where a task goes when it cannot close. 74 have gone there, and each archive entry records what was true at that moment and what was proven afterwards, in exactly one of four buckets. For two of the 74, the gate was green on the day. Twenty-one were later proven by a successor item that checked the final state. Thirty-four were closed by what my system calls Kenntnisnahme, acknowledgement: a founder note of at least forty characters saying that I knew, after which the item leaves the scope. Seventeen left with nothing at all.

Tap to enlarge.

That is the second door. Acknowledgement is not proof, and my own archive tool says so next to every one of the 34: a note from me, no gate evidence. I built a gate that could not be passed by an agent saying "done", and then I built a door next to it that opens when I say "I know". Here is how that door was mostly used. On 3 September the list of unverified archives was 29 lines long. It stood at the top of every new session, and I had stopped reading it. So I decided, once, that all 29 were acknowledged, and a script stamped them, with the old archive reason as the forty characters. Two more followed on 5 September, and three on 7 September carry the reason, translated: "the list of unverified archives is noted; no recurring notification wanted." The door was built to give the list a reader. Its first big use was to make the list stop asking.

Does the door stay? Yes. It is logged, who and when and why; it opens no earlier than a day after the item was archived; and every item that went through it is counted at the start of every new session, where the seventeen that went through nothing turn the line red. The reader of that line is me, every morning, and for the first weeks the honest answer to what I did with it is: I asked it to be quiet. One more thing the numbers do not settle: a second AI session is independent of the first by procedure, not by nature, and may share its blind spots. Still, a door with a log and a reader is a different object from a folder of things that had been said so.

What I Switched Off, and What I Refused To

The objection I have been circling deserves its own paragraph. 484 work items in 42 days is eleven and a half a day, for one person; 107 of them in four days; 246 untouched for two weeks. The disease may be the spawning of tasks, not the checking of them. And why not simply read the changes myself, as a reviewer would? I did, before 2 August; that was the folder of things that had been said so. I think the objection is half right. The apparatus did not open those tasks; I did, or my agents did on my instruction, and a control that costs eight minutes per pass on eleven tasks a day is a different animal from the same control on two. That goes on the October list as well, and it is not a change to the apparatus.

My guess, ahead of the data: the verifier earns its keep and the round mechanism does not. On 3 October I will know whether I was right, because on 5 September I decided three things and wrote them down as a decision record, so that I could not drift back.

Until 3 October, nothing in the apparatus gets rebuilt; the only permitted change is the repair of a test suite that is actually failing. Every session gets one verifier round, a second only if the first found something serious, one mutation probe per new mechanism, one run of the chain. And an after-measurement with targets written before any data: median tool calls per work-item session at or below 400, wall clock at or below six hours, at most 45 percent of sessions blocked.

Eight days in, measured the same way: eleven work-item sessions, median wall clock 6.4 hours, one of the eleven blocked, median tool calls 737. One of three targets met, and on the one number that measures how much the agents churn, the reform made it worse: 531 to 737. Against today's recount of the same window, 7.9 hours, 514 calls and 61 percent, two of the three gains shrink and the third reverses further; I keep both baselines because the gap between them is a finding of its own. The targets are what I hope for. The kill rule I wrote was weaker: the reform would be declared ineffective only if none of the three figures beat its 5 September value. Two of the three already did, after eight days. The rule was satisfied before it could ever fire, which makes it a ceremony too. It goes to the top of the list for 3 October.

What I did not switch off: the verifier, the fingerprint, the rule that a FAIL cannot be paraphrased into a PASS. Each has a real catch behind it. A check with a reader and a price is a check. A check with neither is a ceremony, and I had been holding ceremonies with the lights on.

The next control I build gets three lines written before its first line of code: who reads its output, what one pass costs, and the number that turns it off. This one had a reader, no price, and a number that could not fire. It took 33 days and a plan that said "fill in" to notice.

How I counted: work items, archive entries, verifier rounds and final reports from my own records; the ten FAILs are every tenth file of the 107 in alphabetical order, checked against the code at the commit each report names; stop-gate runs and session lengths from my tool-usage log; commits from the git history of my workspace. The stop-gate sum is 709 runs with a logged duration, 147 seconds, between 15 August and 5 September. The verifier estimate is 295 rounds in work items dated 20 August to 5 September, at the 7.9-minute median of four hand-timed rounds (5.4 to 9.2). The 260 hours are the 28 work-item sessions that started in those days, first action to last. The 5 September session figures are the ones I decided on that day, over 23 sessions; today's recount of the same window finds 28 sessions with a median of 7.9 hours, 514 tool calls and 61 percent blocked, and I have kept the decision-day figures in the text. All other counts were taken on 13 September 2026 and are reproducible from those files.

---