My Tests Were Green Because I Wrote Them

On July 20 my bot's comment engine looked at thirty-four posts and threw away every one of them as too old. None of them was old. An adversarial audit of that exact function had run twenty-one minutes earlier and found four other things.

The log line was two fields long: class_c_found: 0, class_c_skipped_stale: 34. Thirty-four foreign posts examined, ten at most per author, all discarded. The list those authors come from held over two hundred posts from that same week.

Drafted with AI assistance; however, every number, failure, and opinion in here is mine

There is one person in this company and it is me, so the bot is the entire marketing department. Part of it writes its own posts; another comments on other people's, from a list of authors I curate by hand. That second part had gone silent while every instrument I owned said it was fine. (The standstill I wrote about last time began three days after this evening. Different failure, worse week.)

---

Thirty-Four Rejections, All for the Wrong Reason

The age filter existed because of an embarrassment. Action #66 in the bot's log had put a comment under a Bluesky post from January 27, 2025 — eighteen months old, by an author who had not posted in 338 days. It looked like what it was: a machine with no idea what year the thing it was answering came from.

So I built the gate that evening. Two timestamps per foreign post: the one the author's own client wrote into the record, and the one the network's index recorded. Take the more conservative, refuse anything missing, unparseable or older than the limit. Thirty-eight tests for that one gate. Then an audit by a different model, adversarially prompted, over the same code: four findings, one a real blocker about trusting a single timestamp. Fixed nine minutes later, five more tests, suite at 1,174 green.

The scheduled cycle ran at 19:30 and found nothing at all.

My code read record.createdAt and post.indexedAt. The library exposes those fields as record.created_at and post.indexed_at. Both reads returned None on every real object. I had written the first of the two thirty-one minutes earlier, inside the gate itself; the second had been sitting in the discovery path since I wired it up four days before.

And the counter lied about it. None is not "too old": the code has a separate rejection reason for a missing timestamp, and it fires correctly. But every recency rejection increments one field, and that field is named class_c_skipped_stale. So the log told me thirty-four posts were too old. Thirty-four posts had no readable date at all. One number, two meanings, one name — remember this, because the second half of this piece is the same mistake at a larger size.

Before the gate existed, a missing timestamp had quietly defaulted to "now." That is why action #66 happened: not because no age filter existed, but because the date was never being read.

Zum Vergrößern antippen.

A Mock Agrees With Any Name You Invent

None of what follows is a story about carelessness, and I would rather it were. Every one of those tests ran against a mock. A Python MagicMock fabricates any attribute you ask it for. mock.indexedAt works. mock.indexed_at works. mock.please_explain_yourself works, and returns another mock, cheerfully. A test that sets indexedAt on a fake and asserts my adapter read it proves my test and my adapter agree. It says nothing about the thing on the other end.

Here an experienced reader is objecting, correctly: the standard library has had create_autospec and spec_set= for years, exactly for this, and I did not use them. That is on me. But look at what the cure does. It builds the mock's list of legal attribute names by importing the real class and reading it — the fix works by fetching truth from the other side, which is this whole lesson wearing a different hat.

Nothing else caught it either. Dry-run mode swaps the adapters for stubs before any real object shape is in play. The type checker runs in strict mode here, but the library import carries a type: ignore[import-untyped] and the function signature reads (post: Any, record: Any). Strict typing over Any is a strict promise about nothing.

And the existing tests were wrong in my favour: fixing the adapter meant editing two test files to snake_case, because they had encoded the defect rather than caught it. A test written from the same misunderstanding as the code will pass forever and mean nothing.

My commit message that evening — I write them in German — contains a stutter I left in, translated: "full suite: 1178 green (previously 1178… previously 1174, +4 new tests)." That is the count after the fix; when the thing fell over it was 1,174. I could not keep my own numbers straight in the half hour I spent writing down what the numbers had failed to tell me.

The Two Bugs I Was Not Looking For

Once you have found one attribute-name bug, the honest move is to assume it is a species and not an individual. I read the whole adapter for the same pattern and found two more, both dating to the day it was written on July 11.

The mention reader used notif.indexedAt, so every incoming mention had been stamped with the fallback value — the moment my code looked at it, rather than the moment it arrived. The engagement reader used likeCount, replyCount, repostCount, where the library spells them like_count, reply_count, repost_count. Every engagement snapshot for every real post had come back 0/0/0.

That is not an inaccurate number. It is an absent number wearing the costume of a measurement. Nine days of it, in a table I had looked at. A zero meaning "not read" is indistinguishable from a zero meaning "nobody cared" — entirely plausible for products that sell in the low single digits. What should have tipped me off was not the zero. It was the total absence of variance. Nothing real is that steady.

One live call surfaced one bug and paid for two more, because those two only became findable once the first told me what to look for. No amount of internal testing produces that. I checked the second platform's adapter for the same pattern and found none: that library returns plain dictionaries, and had been correct all along for no reason I can take credit for.

Then I Built an Instrument to Check My Work

Two weeks later, a different project, same failure one level up: this time the checking tool took its truth from the thing it checked.

I am building a linter that judges the layout quality of generated documents — half-empty pages, orphaned headings, graphics that overrun their column. It is written in JavaScript, so the code below changes language. Before any product code got written, the measuring apparatus had to be finished: 267 numbers printed in the tool's own specification, each pinned to a value from a real run; one set of scripts that re-derives them; a second set that checks the first set can actually fail; a runner that prints one verdict over the lot.

On the morning of August 3 the runner printed VERDICT: OK and exited 0. Called on its own, the number checker reported "freshness check: active" and 267 of 267 passing — against a stamp file from the day before.

Then I handed the same tree to a verifier running in a fresh session of a language model, with no memory of having built it. It came back with VERDICT: FAIL and eleven findings: one blocker, two high, seven medium, one low. Five are below; the blocker turns up in the last section; the rest are narrower versions of the same habit.

Zum Vergrößern antippen.

Five Ways the Instrument Flattered Itself

The freshness check took its idea of "when did this run start" from a stamp file that ships alongside the very data it validates. And when that stamp had no start time in it, the comparison became modified_time < undefined, which in JavaScript is always false. A raw file backdated to 2020-01-01 passed.

The guard checker read from whatever directory you pointed it at and wrote its report to its own. The prescribed negative control is to run it against an empty directory and confirm it screams. It screamed. Then it wrote "NOTHING CHECKED" over the delivered evidence file. I know the exact moment, because that file's hash changed under it: 423b1451 became 8a62ad35. The control destroyed the record it existed to protect, and left a receipt.

The check for "does every guard have a real failure path" was a regular expression looking for process.exit( followed by any number, with no look at the condition attached. An exit(2) inside if (argv.length < 1) satisfied it: a script that only knows how to complain about being called wrong counted as a script with teeth. The file's own header promised the condition had to read a measured value. That promise is gone from the header now — it was never implementable from source text, and saying so was cheaper than continuing to claim it.

The runner had been verifying a copy of the contract document instead of the canonical one. The two had been byte-identical for nine consecutive rounds. That is diligence, not a property. A change to the real contract would have walked past all four checkers without a mark.

And the headline number went too. "267 of 267 checked against raw data" was formally true and materially misleading: 241 were held against a raw value, the other 26 only against a relation between values. Two things, one name — the same shape as class_c_skipped_stale, two weeks and a different language away.

A check you have never watched go red is a claim, not a check.

"Integration Tests Are Flaky"

They are, and I am not asking for a hundred live calls in continuous integration. One is enough: once, per foreign interface, recorded.

For the library that broke my adapter, that verification was a few lines in a Python shell: import the models, print their field names. Not the network — the installed package, the same one my broken code imported from. A weaker oracle than a captured response off the wire, and still enough: it was the one source in the room I had not written.

Get a second model to audit it, then. That audit ran twenty-one minutes earlier, adversarially, over that exact function, and found four other things including a blocker I am glad to have paid nine minutes for. It did not find this. What that proves is narrower than it looks: the auditor could have opened the installed library and did not. A reviewer inherits your scope unless something forces it out, and mine was scoped to my code, which is where I had already looked.

What Changed in How I Work

Four things, all boring, which is a good sign.

Every adapter to a third-party library gets one verification against a real object before it counts as verified: field names dumped from the package, a captured response, anything from the other side rather than from my head. The mocks stay. They are cheap and they hold my code to my assumption. They are not evidence about the world, and that sentence now sits in the file.

Every checker has to name the condition that turns it red, and somebody has to have watched it turn red. This is where the blocker sat: the reassuring pass had only ever been produced inside a temporary directory the operating system creates without spaces in the name — precisely the failure that stage exists to catch. It now runs in two locations, one with a space in the path. To prove the second arm works I put the old broken line back into a copy and ran both: same files, same minute, pass in one and failure in the other.

Nothing a checker depends on may be supplied by the thing it checks, and it must read the canonical file rather than a copy. Run identity now comes from the calling process, not from a stamp travelling with the data.

And a counter that counts two things is two counters. Both halves of this piece are that defect: a name promising one meaning while a number carried two.

What I did not do was ship the product and fix the instrument later: production code waits until the measuring apparatus is closed out. That delayed the product by an amount I did not measure — which, in a piece about measurement, I am obliged to say out loud rather than round up into a number.

One thing is not closed. The verifier that failed my instrument listed eight gaps of its own; two are that the diff it was handed was zero bytes long, and that around fifty-five of the scripts were never given to it. It also never executed anything: it read source. So the audit that found eleven problems was itself working from a partial view, and said so. Which means I do not currently know whether the instrument is right. I know it is less wrong than it was on Monday morning, and that is a different claim.

The longer set of these rules is in The AI-First Operating Playbook, with the rest of what I paid retail to learn. This part is free, and it is one sentence: a passing suite tells you your code agrees with you.