My Second Opinion Was Dead for Four Days. At Least.

I keep three separate AI models to audit my work, on the theory that they will disagree with me in different ways. On July 28 I sat down to score them against a list of bugs I had written in advance. One of the three had been refusing to answer since the 24th, and I had not noticed, because in those four days I had not asked it anything.

Exit code 7. No output file, no partial result. The wrapper I put around that model could not prove which model had answered, so it threw the answer away rather than hand me something of unknown origin.

Drafted with AI assistance; however, every number, failure, and opinion in here is mine

I would build that behaviour again. What it exposed was not the outage. It was that I had built a second opinion, stopped consulting it, and wrapped it in machinery healthy enough to keep failing quietly on my behalf.

One word first, because everything below turns on it. A runner is not a model. It is the model plus my wrapper script: what I send, what shape I demand back, what I refuse to accept.

---

Exit 7, No Output, Nobody Asking

The cause was small and stupid in the way these always are. The vendor's command-line tool reports which model handled a request in a usage field. My wrapper compares that field against the string grok-4.5 and aborts if it does not match, so a silent downgrade can never pass itself off as the review I asked for. At some point the tool started labelling the same model grok-4.5-build. Equality failed. The wrapper did as it was told and produced nothing.

The repair was one condition: compare against an anchored prefix instead of the whole string, and name the value it actually saw. I made the check fail on purpose twice before I believed it was fixed — once by restoring the old bug, once by loosening the prefix into a plain substring match, which would cheerfully accept a model called not-grok-4.5-at-all.

The test suite could not have caught this. Its fake command-line tool returned the key grok-4.5, hard-coded, because that is the key the check expected. The fake and the check were built from the same sentence in my head. I have written about this species of tautology before, and I still shipped another one, in the file whose entire job was to detect an impostor.

Now the part that should have gone in the title.

Twenty runs sit in that log on the evening of July 24, all successful, and I anchored "four days" on the last one. Every one of the twenty was the test suite — the one driving the hard-coded fake. That is not evidence that anything worked. Filter the log down to real reviews and the last success is July 17. Eight real calls on the 18th and 19th exited 7. I labelled one fennec-page-code-diag, which is what I call a file when I am annoyed and diagnosing, then blamed capacity and moved on.

Between that last real success on July 17 and the next one on July 29 sit twelve days in which no real review of mine came back. How much of that was this bug I cannot say: exit 7 is what the wrapper returns for any validation failure, and the error output went to temporary files that are gone. Four days is the floor, twelve is the span, and the number I put in the title is the one I could defend without doing any work.

That is a counter mixing test runs with real work, which is the exact defect I complain about later in this piece. I did not catch it. I caught it because I ran a draft of this article past an independent check, which is either reassuring or the joke writing itself.

I also was not away. During those four days I ran ten audits with a different model, including a morning on the 27th spent watching it reject a third opinion twice in a row. The dead one lay in a directory the whole time. I fixed it that evening, which is why it appears in the scores below at all.

Then I Wrote the Answers Down First

I wanted to know which of these runners actually finds things, and I had never measured it. Three opinions feel like diligence. I had been running on the feeling for weeks.

So I built a file to fail. Seventy-one lines of Python imitating a daily gate for autonomous social comments — the thing deciding whether my marketing bot may reply to a stranger's post today. Close enough to my real system to be interesting, entirely synthetic, so nothing real leaves the room. Then I wrote down what was wrong with it before anyone looked.

Five defects planted on purpose. A membership check testing substrings against a comma-joined string, so an author named ann gets in on the back of joanne. A greater-than where a greater-or-equal belongs, worth exactly one comment over the daily cap. A catch-all except Exception returning True, converting any unexpected error into permission to publish. A timestamp carrying a timezone compared against one that does not, which raises an error, which lands in that handler, which returns True. And a check that publishes before it books the comment.

Two more were accidental, found while writing the file and left in on purpose: an off-by-one admitting posts up to nearly sixty-one days old against a sixty-day rule, and a future-dated timestamp producing a negative age and sailing through.

And one decoy: a cap calculation that reads like a race waiting to happen and is in fact deterministic and correct. Anyone who reports it produces a false alarm and I get to count it.

Same prompt text to all three: line number, severity, one sentence, ten lines maximum, no preamble. I am not measuring how well these models write. I am measuring whether they find things I already know are there.

Four caveats before the numbers, because a comparison that hides its asymmetries is advertising. One runner could not be handed the file the way the others were and got it inside the prompt instead — that was the one that won, which is the direction I would least like a handicap to run. Each ran once, so any one-finding gap sits inside the noise. I planted the defects, wrote the decoy and graded the answers myself, unblinded. And the prompt text being identical is not the same as the request being identical: one of these went through a script of mine that demands more than the prompt asks for. Hold that thought.

Everyone Found the Easy Three

All three found the first three defects. All three, no partial credit.

Nobody took the decoy. Nobody produced a single false alarm, which surprised me more than the hit rates did, because what I had braced for was a wall of plausible nonsense about defensive programming.

The scores diverged only afterwards: six of seven, five plus a caveat I will come back to, and three.

Tap to enlarge.

Look at where the agreement sits. The three that everybody caught all fit inside a single expression; you can see the whole of each without moving your eyes. Two defects genuinely span two places in the file. The non-atomic check was found by two of the four runs. The timezone comparison was found by none.

That last one deserves naming. A timestamp with a timezone meets one without, Python raises a TypeError, the catch-all swallows it and returns True, and True there means this post is fresh, go ahead and comment. All three found the catch-all. None connected it to the thing that would walk into it. They found the sink and missed the source, which is not blindness and is not much better.

On this file, at least, consensus was cheapest exactly where it was least useful.

The Same Model, Two Answers

Here is the finding that changed how I use all of this, and it is not about a vendor.

One model appears twice in my results. Five of seven through my review script, four of seven called directly with a loose prompt. Same model, same seventy-one lines, same evening, minutes apart.

That is the thought I asked you to hold. The only difference was what I demanded back. My script forces a response shape: findings, then a section called coverage.gaps for whatever it could not determine. The direct call asked for findings and stopped there. Given a place to put uncertainty, it found the non-atomic check it had otherwise missed, and it wrote this (translated — I work in German):

No definition of timezone or UTC for day_key and created_at.

Not a finding. An admission. And that made it the only runner in the test that touched the defect none of them reported, because asking a model exclusively for errors gets you only what it is willing to call an error, and everything it is unsure about falls off the table on the way to you.

I thought I was measuring three models. Part of what I was measuring was the shape of my own question, and I do not know how large that part is, because I would have to run this many more times to find out. One finding is inside the noise. The direction is not: uncertainty has to be given somewhere to go, or it does not arrive.

That change was free and I had to be shown it by the runner that came second.

What the Outside Actually Bought Me

The cynical reading is available here, so let me spend the rest of this on why I do not hold it.

Those twenty test-suite runs on the 24th belonged to a feature I was finishing that night: letting one of these models read a sanitised copy of a repository, secrets and private directories stripped from the copy first. I audited it myself first, adversarially, in the mood where you assume you are the problem, and found four real things — among them a pointer back to the main repository that still exposed the full history, including secrets committed once and deleted later.

Then I handed the whole thing to another model. It came back with six findings it called blocking, meaning ship this and something leaks. Verdict NO-GO.

And the one that mattered was not a bug. It was the premise. My decision rested on the assumption that pointing the tool at a sanitised directory limits what it can read. The vendor's own sandbox documentation, sitting on my disk the entire time, says the profile I had chosen grants filesystem read access everywhere. Not "the working directory." Everywhere. The sanitising layer I had carefully built was not incomplete. It was decorative.

The feature is still in the repository and still switched off. I keep it as furniture.

It happened again with the next integration, at a worse ratio: one blocker found by me, ten by the outside. The second round failed too, on damage I did while repairing the first — my fix for an exposed key moved the key out of one visible place into another.

Four found by me, six by the outside. Then one by me, ten by the outside. More, obviously. But mostly a different kind of thing. I audit implementations well enough; I cannot audit my own premises, because I am looking through them.

The Part I Have Not Fixed

Here is what my logs say about which second opinion I actually reach for. Over the ten days when all three existed side by side, counting only real work and discarding test-suite, probe and benchmark runs, because by now we both know what a mixed counter is worth: 81, 24, 21.

Tap to enlarge.

Some of that gap is reachability rather than preference: two of those ten days are the outage and the build. Most of it is not. And the awkward part is that my own scorecard ranks the three in exactly the order I was already using them, so the measurement I ran to challenge my instinct agreed with it. That is less flattering than it sounds. It means the test that could have told me something was the one I spent weeks not running.

One more line in those logs bothers me. A run on July 27 where I asked for one model version and an older one answered. Exit code 0: success, by every signal a script checks. I know only because that log stores what I asked for and what replied in two separate fields. Which leaves a question I cannot close. I verified provenance for one runner and demonstrated its absence in another, in the same week I scored all three.

So: my fakes now take the attribute names they are allowed to answer to from the real class, instead of inventing whatever they are asked for. Every guard has to name the condition that turns it red, and somebody has to have watched it go red. Two of my three audit prompts now demand a gaps section. The third still does not, which I noticed while writing this paragraph.

What I wrote down as the fix was a monthly run against a file with known defects. It sits in my own report as a recommendation, with my name on it twice under two different labels and a status of "open, founder decision", which for a company of one person means nobody. I wrote the finding. I wrote the fix. I did not build it.

And a monthly canary would probably have missed this outage anyway. The thing that would have caught it on day one was already sitting in the log: eight real calls exiting non-zero, which nothing was watching, because I had built a second opinion and never built anything to tell me when it stopped answering.

Eight days since I wrote that recommendation. I will put the number in the next one of these, whatever it says.

Three opinions are not a measurement. One of mine went twelve days without answering, and the number I first wrote down was four.

---

I write about running a one-person company on AI agents. The longer version of how these are wired together, including the parts that worked, is in Orchestrating AI Agents.