I Ran the Same Audit Nine Times. Four Missed the Hole.

Nine runs on three files I wrote myself. Same prompt, same data, same morning — the only variable was which model read it, and what four of them missed was the one entry that wasn't there.

In the first two weeks of August I hit the weekly limit on my Claude subscription more than once. I run a one-person company almost entirely through agents, so that limit is not an abstraction. When it lands, the shop is shut until it resets.

Drafted with AI assistance; however, every number, failure, and opinion in here is mine

So I asked the obvious question: which of my agents can I move to the cheaper model without losing anything? There are eight of them. A strategist, a programmer, a bookkeeper, a brand critic, an operations watcher, a researcher, a science writer, a marketer. Moving two or three of the quieter ones down a tier felt like the kind of tidy Sunday saving that costs nothing.

Then I measured, and the question fell apart in my hands.

The Question I Asked Was Wrong

The measurement covered 65,989 assistant turns across 1,461 transcripts, thirty days, every repository I worked in. Everything below is weighted: a turn's tokens multiplied by that model's published list price, so an expensive model's turn counts for more than a cheap one's. Nothing is billed per token on a subscription, so this is a share of pressure on the limit, not a bill.

I wrote the tool that morning, and I want to be honest about what it bought: a morning of work to discover that my original plan would have saved almost nothing.

Main threads, meaning everything that is not a subagent, account for 67.2 percent. My programmer agent takes another 19.3. Generic unnamed helper agents take 7.8. Every other named persona I have, all seven together, comes to 4.0 percent, and the remaining 1.7 is agents belonging to a second venture plus built-in tooling. My operations watcher, the one I was most confident I could downgrade, ran 27 turns in thirty days. It rounds to zero.

Tap to enlarge.

Two more numbers from that run stayed with me. 97 percent of all tokens I consume are re-read context — not new text a model writes, but the same conversation handed back to it, turn after turn. And turns in the 201st-to-400th band cost about three and a half times what turns in the first twenty-five do. The expense isn't in the thinking. It's in the remembering.

None of which answers the question I actually needed answered, which is whether the cheaper model is worse. So I built a test.

A Service That Wasn't There

Three test pieces: a Python module for code review, a decision memo for document review, and a pair of operational JSON files for a target-versus-actual check. I planted defects in each, wrote the answer key into a file outside the test material, and froze it before a single run started. Then I ran both models against each piece with identical prompts, fresh context, and the real system prompts my agents use, checking afterwards from the subagent transcripts that each run had actually used the model I thought it had. Two independent instances graded the reports blind, without knowing which arm produced which. One of the two graders was a different vendor's model, so that a Claude model was never the sole judge of a Claude model.

The third piece is the one this article is about. Nothing in it is a real incident: two files I wrote myself, modelled on my real monitoring. An inventory listing seven services, and a synthetic snapshot of what supposedly ran last week, listing six.

The missing one is called accounting-backup. In the inventory it is flagged critical and expected once a week. In the snapshot there is no entry for it. No error, no failure count, no timestamp, no zero. Nothing.

I put that hole there on purpose, which means I know exactly how visible I made it. Not very. Because the snapshot also carries a helpful note of its own, which I wrote to be exactly as true and exactly as useless as such notes are in real life:

All services contained in the snapshot were checked.

That sentence is accurate. It is also the wall the hole is hiding behind.

Four Reports Never Noticed

Nine runs on that fixture: four on Opus 5, five on Sonnet 5. The first four were two per arm and went to the blind graders with everything else. Then I added five repeats, three on the cheap model and two on the expensive one, and scored those five myself, on one question: did the report name the missing service. That is the weakest joint in the whole test, and I would rather point at it than have you find it.

Opus found the hole in four runs out of four. Sonnet found it in one out of five. Fisher's exact test, two-sided: p = 0.048. Thin, and I will come back to that.

What I did not expect was the shape of the failures. The four reports that missed it are not bad reports. All four flag that the inventory declares a calendar week while the snapshot never says which window it used, so the run counts cannot be matched to the declared period; three of them prove the shift from the timestamps. All four flag that a critical mail service logged 168 runs but only 166 clean exits. One of them warns, in so many words, that treating the matching totals as proof would be a false green.

Then it stops. Two findings, and done. All four of them: exactly two findings.

The one Sonnet run that did find the hole reported three. The four Opus runs reported six, nine, nine, and ten.

Two objections come up here, and both are answerable from the same morning's data. The first is that the cheap model simply writes shorter reports, and short reports find less. It doesn't survive the other fixture: on the decision memo, the same model found nine planted defects out of nine, in every run. It is not generically terse. It went long where the defects were in front of it and stopped early where the defect was an empty space.

The second is that the long reports were just noisier — more findings, more noise, one of which happened to be the right one. I had planted four decoys in that fixture for exactly this reason. Neither grader logged a single decoy hit on it, in any of the four graded runs. They did split on the extra findings the long runs produced beyond the answer key: one grader judged them all genuine, the other judged about half of them noise. So the long runs were not clean, but they were not padding either, and nobody walked into a trap.

And then there is run seven, which I keep rereading. It had to open the inventory to know which services were critical at all, because that is the only file carrying the flag, and it correctly left out the one job that isn't critical. Then it built its list from the other file, the one the sixth service was missing from, and announced that its finding covered "all five critical launchd services." It named five. The inventory declares six.

That is not carelessness. That is a report about a missing entry, missing the entry, in the same way.

Tap to enlarge.

The Line Isn't Where I Thought

Both models scored 100 percent on that decision memo — nine defects out of nine, every run, both arms, including a citation that says the opposite of what the text claims it says and a proposal that violates a binding internal rule. Opus burned 4.4 times the tokens to reach the same score. On the code review the two came within four points of each other, and the grader who separated them at all was the one docking an Opus run while giving a Sonnet run a clean sheet. Close enough that I would not spend money on the difference.

So the expensive model is not generally better. It is specifically better, at one thing.

Both graders were blind to the arms and neither ever names a model. What they did, independently, was single out the same fixture as the only place where the runs came apart: the one where the finding was an absence. The wording that follows is mine, not theirs. On my fixtures, the cheaper model reliably reports what is there, and does not notice what is missing. Where the object of the review lies fully in front of it, in a document or a function or a diff, it is an equal. Where the finding is an entry that should exist and doesn't, it walks past.

That is nine runs on three files I wrote. It is enough to route on, and not enough to publish about a model — and the reason it is enough to route on is that the two mistakes cost wildly different amounts. A false alarm costs me a minute of triage. A backup that quietly stopped running costs whatever it was holding.

It was also enough to show me that the axis I had been routing on for months was fiction. I assigned models by persona, as if my bookkeeper were inherently a harder job than my programmer. Personas are a useful fiction for tone and permissions; they are worthless as a boundary for this. The real boundary cuts straight through them. My ops watcher, my bookkeeper and my marketer all spend most of their time reconciling something against an inventory, a ledger, or a stated expectation. That is the exact shape the cheap model walks past, and together those three are under one percent of my spend. Downgrading them would have traded my most expensive class of error for under half a percent.

The Answer Key Was Wrong, and the Model Caught It

I had planted four decoys in that fixture — things that look wrong and aren't. One of them was the executive briefing: last run recorded on Monday, 11 August. Correct, I wrote in the answer key. Nothing to see.

11 August 2026 is a Tuesday.

The briefing is scheduled for Mondays. There is exactly one Monday in that window and no run on it. So my decoy was a real defect that I had labelled correct, and my answer key was wrong at the moment I froze it.

Three of the four Opus runs named the Tuesday. No Sonnet run did. And the grader that first told me, the one that went and checked the calendar instead of accepting my label, was a model reading the reports. Not me.

There is a second one like it, and it is worse for me. That decision memo contains a block quote attributed to my own research corpus. The quote does not exist. I wrote it as scenery and then forgot it was scenery. A grader noticed afterwards. Not one of the fourteen graded runs, on either model, had asked whether the source it was being handed was real.

I keep a file of lessons about oracles: about how, when the thing that defines truth and the thing being tested come out of the same hand, the test mostly measures the hand. I wrote that file. I still froze an answer key with an error in it and a fabricated quote inside the exhibit. What saves the result is that the error made the test stricter than designed rather than weaker, and the separation between the two models is there with the correction and without it.

What This Actually Buys

The decision was to route by the shape of the finding rather than by who is asking. My programmer agent now runs on the cheaper model by default, with the expensive one made mandatory wherever a missing entry is the finding: audits, four-eyes sign-off, independent verification, security, anything reconciled against an inventory. The reconciling personas stay where they are.

Implementing that meant changing five places in lockstep. My own documentation said four. The validator found the fifth, which had been missing from my list of places to change since I wrote the list.

The expected saving is about ten percent of consumption — expected, not measured, because the health records my agents write carry no model field, so I cannot attribute it afterwards. On 19 August my weekly limits drop by roughly a third, because a promotional uplift I had been living inside since May expires. I am close enough to the ceiling that the comparison is fair, and ten percent does not cover a third.

The lever that would cover it is the one I had labelled wrong.

I have been calling that 67.2 percent band "the interactive session" — in my notes, in the decision record I ratified this morning, and in the first draft of this article, where I wrote that is me, typing. It isn't. The label in my own tool means "not a subagent", which is not the same claim. Split by entry point, 34.6 percent of everything is headless runs that start themselves on a schedule, and 32.6 percent is me, typing. Roughly half of what I had been calling my own behaviour is automation I wrote months ago and stopped seeing.

I should be clear that this is good news, which is the uncomfortable part. It is a third of my consumption sitting in scheduled jobs I could shorten without changing how I work at all — a bigger and easier lever than the one I spent the morning on. I only found it because someone checked the label, and I only had it checked because the article needed the number to be right.

Two more things I owe the record. I also built a change to move my second venture's daily runner onto the cheap model, verified it, and then threw it away — the verification is what killed it. Those runner branches don't replace the daily cycle prompt, they prepend a block to it, so switching them would have switched the entire daily cycle, including the parts that publish to a shop and post to social accounts, while looking in the diff like it only touched the runner. The scheduler runs the working tree, not the last commit. It would have gone live at 15:00 that afternoon.

And the cheaper default is fail-open by construction. A forgotten model parameter now runs an audit on the wrong tier and returns green. My guard against that lives in a prompt, not in a mechanism, and a guard that lives in a prompt is a hope with good posture. Building the mechanical version is the next job, and it is not done.

The number the whole thing rests on is four out of four against one out of five, on one of three fixtures — and the other two separated nothing, so the p-value has been shopping. What holds it up is not the p-value. It is that the direction held across every repetition, that two blind graders isolated the same task class without conferring, that my own accidental decoy pointed the same way, and that the two kinds of error cost such different amounts.

You cannot see an absence by looking harder at what is present. On the evidence of the last two weeks, that holds for the person writing the prompt as well.