My Test Documents Had No Pictures in Them

I sell guides made of HTML. I also build the tool that checks whether their pages came out right. It had twenty-one test documents, and not one of them contained the thing I put on 40 of the 74 HTML pages I sell. If you have ever shipped a checker, a linter or a rule engine, this is a story about the material you checked it against.

Drafted with AI assistance; however, every number, failure, and opinion in here is mine

I build a tool called breaklint. It has no window; I run it by typing its name and a filename. The guides I sell are written as web pages and turned into printed pages at the end, and that last step is where things go wrong. So breaklint does that step itself, with real page breaks, and then measures the page that would actually come out. A heading stranded alone at the foot of a page, so the reader turns over to find out what it was about. A paragraph whose last line is a single word. A page that stops a third of the way down and leaves the rest white, in a document somebody paid for. It is for people who ship documents made of HTML: reports, manuals, books. The guides I sell. Rather than describe what it prints, here it is, from this morning, against page 2 of one of those guides.

warn  layout/half-empty-page  page 2
  measured   0.59 fill ratio; threshold 0.6 fill ratio (uncalibrated)
  detail     Page 2 is 59.0 % filled; threshold 60 %.

That block is a finding: one complaint about one page, with the number it measured and the number it compared against. Fifty-nine percent full, against a line drawn at sixty. Nothing is wrong with that page. I picked the sixty, in an afternoon, and the tool prints the word uncalibrated next to it every single time, because that is what it is.

The same tool, on the same day, found something that was genuinely wrong in a document I had written, proofread and sold. It took me until the end of this piece to run it there.

On 27 August I pointed the same tool at another page of my own, and it did not print a finding at all. It ended in exit 3, which in my own table of exit codes means the checker crashed. That document was mine, it was not written for the test suite, and that turned out to be the distinction that mattered.

The Test Documents Were the Part I Made Up

A checker is two things: the rules, and the documents you check the rules against. I had been careful about the first one for a month. Four hundred and fifty-seven automated tests, sixty of them driving a real browser, all green that morning.

The documents came out of my head. I sat down and imagined documents that would break. A document whose font never arrives. A document that quietly removes my measuring marks while I am still adding them. A document that lies to the browser about how big things are. Twenty-one files, every one of them written to make a rule fire, and every one of them written by the person the rules also belong to. In the trade they are called fixtures, which is a fair name for furniture you built to fit your own room.

Then I counted what my own product pages contain. Across the six guides I sell, a buyer opens 74 HTML pages, and 40 of them carry a chart drawn directly into the markup rather than pasted in as a picture.

The command that found it took one: grep -l "<svg" tests/fixtures/*.html. It printed nothing, which is how grep says it found nothing. <svg is what a chart drawn into the page looks like in the file.

A command that prints nothing looks exactly the same when the path is wrong, so I pointed the same one at the guides instead. It printed forty filenames. That check cost four seconds, I nearly skipped it, and skipping it is the entire subject of this piece.

Two of the twenty-one fixtures do hold a picture, and that command does not find them, because both link out to an image file instead. That is not a technicality.

Tap to enlarge.

So I did not forget figures. I ship them constantly, in the thing I charge money for, and left them out of the material that decides whether the tool works. The gap is not ignorance. I built the test files and the product for different purposes, and only one of those purposes was "make the suite go green".

It Was Talking About Itself, and Addressing Your Document

The crash was not subtle. A two-page document, one harmless chart sitting well inside its frame.

Before the rules run, one step walks the rendered document and writes down every box and every line height. On this document it gave up on the chart with a complaint no rule had been taught to recognise, and an unrecognised complaint stops the run, on the theory that a checker should never quietly skip something it does not understand. So it stopped.

I took the fatal part out, and the run then failed a second way: too little of the document had been judged to call the result anything. The tool has a code for that too.

Both answers were statements about my tool, delivered as verdicts on somebody's file. The first reads as your document broke me, the second as your document could not be judged. Both of them meant this build cannot do that at all. I gave the second one an honest fix: a measurement my build cannot take now drops out of the sum instead of counting against the reader's file.

Same Commit, Same Night

Two weeks earlier I had fixed something worse and written down the lesson in the same sitting. I only noticed that this week.

The tool asks itself whether a person started it directly, and it asked by comparing two file paths letter by letter. Installing it puts a shortcut in the way, so on every installed copy the two paths read differently, the check said no, and the program did nothing at all: no output, no findings, and the exit code that means clean. A build pipeline believes that code. I had shipped a checker that certified every document it was pointed at, silently, on every machine except mine.

The comment I left above the fix reads: It survived every gate because nothing ever ran the tool the way a user does.

In that same commit, at 23:07 on 13 August, I created a file called docs/limitations.md and wrote into it that two of my rules are still only warnings, because no fixture reaches the path where they would raise an error, and a path no fixture reaches is unverified. I wrote the diagnosis and the cure into the same push, minutes apart. Fourteen days later the chart-measuring code shipped with no fixture reaching it at all.

Why the Guard Slept Through It

I run a mutation guard, which is a test for my tests: it breaks each rule on purpose, five ways, six for the two that need an extra case, and demands that some assertion notices every single time. Seventy-seven mutations, seventy-seven caught, across all fifteen rules. The number was honest and it did nothing for me here.

The guard did not run on documents. It ran on measurement records I typed by hand: a box at these coordinates, a page around it. The rules handled them correctly, chart records included. What was broken was the collector, and the guard began after that boundary. No mutation had ever touched it. It finished in 0.63 seconds, because no browser was ever involved.

It was a smoke detector I had tested with a lighter, at my desk, many times. Nobody had ever hung it in a kitchen.

Which leaves the obvious objection: this is one file's worth of work. Write a fixture with a chart in it and move on.

I did. Then I handed the repair to a language model in a fresh context, with nothing but the code and no idea what it was supposed to prove, and it rejected my repair four times running. Every one of its findings was the same shape: text that the browser lays out and no human can see. Text stored as a template and never drawn on the page. Text made invisible in three different ways, each of which still reports a full rectangle to anyone who asks how big it is.

Two of the fixtures I had written to prove the repair had defects of their own. One was supposed to be the case that stays clean, and its content ran off the edge of the frame, so the thing proving nothing was wrong was itself wrong.

A hole you can close in one file was not that kind of hole.

Thirteen Numbers I Picked, and Two I Did Not

The test-document problem has a sibling, and the two are not the same problem. A missing kind of input is fixed by getting documents. A chosen threshold is not, because no quantity of documents tells you where the line goes without somebody's judgement about which side each page belongs on. I could download five hundred freely licensed documents this afternoon, and not one of them would arrive with a human saying whether a page filled to 59 percent is a defect.

Thirteen of the fifteen rules compare a real measurement against a number a person chose, and the person was me. The other two compare against a structural limit. A block that must not be broken and is taller than the page fits on no page, so the threshold is the page. Text outside a frame is not drawn, so the threshold is zero. Nobody picked those, no pile of documents would move them, and for those two a made-up fixture is genuinely enough, because it can prove something that is true by construction. That is the strongest argument against everything I have said. It is correct, and it covers two rules out of fifteen.

For the other thirteen the chosen number does the deciding. Take the sixty percent from the top of this piece. The most a page of solid text can measure is 68.6 percent, because the space between lines is not ink and does not count. The entire band between "as full as a page gets" and "flagged as half empty" is therefore 8.6 percentage points wide, and a typeface set with slightly more air spends that without trying. I know, because I measured the ceiling and wrote it into the rule's own header, and it is why that rule can never fail a build.

I cannot even write down that these numbers are calibrated. The field is typed in a way that accepts only false; setting it to true does not compile. That was deliberate. It is the one piece of this I would defend without flinching.

Three Documents, and the Question I Was Avoiding

This is where being one person stops being a scheduling problem. I built the whole apparatus for outside judgement: where each document came from, whether it may be used, and three separate piles for developing, tuning and one held back untouched. It holds three documents, and the report records zero people who have marked up a page with a verdict. There is a public channel for reporting a page you would judge differently than the checker does, and it has received nothing at all — which says more about how many people use this than about how hard calibration is.

The better question is the one I have been walking around. Why not annotate the pages myself? I have hundreds of them. I could look at each one, write down whether the warning is fair, sign it, and publish the lot.

I had an answer ready that sounded rigorous: it would be circular, because I chose sixty percent by looking at pages, so grading those pages against sixty percent asks the instrument to mark its own paper. That is true of the pages I already looked at and false of everything else. On a page I have never seen, my judgement is a single anchored annotator, which is weak evidence and is not nothing. The real reason is that it is work, several days of it, and I have not done it. Writing my verdicts down first, before the tool speaks, and publishing them for somebody to disagree with, is the honest next step. It is not calibration, and it is not impossible. It is unstarted.

The Documents I Had and Never Ran

Two of those three were written by strangers, and nothing had ever run the checker over them. The tests around them confirm the files have not changed and the piles are correctly separated. Nobody had ever asked the documents themselves a question.

So I ran all three. Every one ended in exit 3. Pages analysed: 0.

The two Wikimedia diagrams failed because my own measuring probe, which runs inside the page, and the browser's own record of where everything sits disagreed about seven of eight measurements, by up to 0.5477 pixels against a tolerance of 0.05 pixels. My own page failed because the archived copy contained the document without the images it points at, and the browser could not decode them.

Three documents, three verdicts, and not one of them was about a document. Two were about my measuring apparatus and one was about a bundle I had packed wrong. The tool printed it plainly: Checked nothing: the run stopped before it could measure 1 document. This is not a clean result.

What the Half-Pixel Turned Out to Be

Before publishing, I gave the draft to four readers in isolation. Three were language models from three different companies, each handed the text and nothing else — no repository, no notes, no knowledge of me. The fourth was a fresh instance of the same kind, run on its own. None of them could check a single number. They could only read.

Two of them, independently, pushed back on the same sentence. Half a pixel against a tolerance of five hundredths does not sound like a document defeating a checker. It sounds like the checker measuring badly. And if that tolerance is one of the thirteen numbers I picked in an afternoon, then two of my three showpiece failures are artefacts of my own arbitrary line, and the essay argues against itself.

They were right, and the tolerance was not the answer.

I went and measured instead of arguing. The tolerance stayed where it was and never moved. Two people were measuring the same table with two rulers, and one of them was including the tablecloth. Three times, in three different ways. It was never a disagreement about the number; it was a disagreement about what was being measured, reported as a scandal because the results differed by a hair. Fixing the comparison fixed the documents.

I should be exact about what that proves, because "I fixed it" is the kind of sentence this whole piece is about. There is now a gate that runs the checker against a document nobody wrote for my test suite, and the build stops unless it comes back clean. I pointed the old version at the same bytes to be sure the gate can go red: it stopped dead, 18 pixels out. That does not prove the new measurement is right. It proves the old one was wrong and this one no longer collapses, which is a smaller claim and the one I can make.

The Run I Should Have Done On Day One

Which brings me to the sentence I had written as this essay's ending, and can no longer write.

The old ending was: documents this checker has successfully measured that were not written to make it pass — zero. It was true when I typed it. One of those four isolated readers took about a paragraph to point out that I own 74 counter-examples. My product pages were written to be sold, not to make anything go green. By my own definition they qualify exactly. Why had I never run them?

I have now. All 74, one at a time, on the current build.

Tap to enlarge.

Sixty-two came back measured, 392 printed pages between them. Twelve did not: nine crash inside the browser, two on that same geometry comparison in a corner I have not found yet, one measured too little to call the result anything. Eight of those twelve carry a chart drawn into the page, against 40 of all 74 — a lean in the direction this whole piece predicts, on a sample of twelve, which is to say a hint and not a finding.

It also found something. On page 2 of the file that tells a buyer how to start, a numbered heading reading Measure the ROI sits at the very bottom of the page with 0.15 of a line of space beneath it. Steps one to three each have their sentence underneath them. Step four does not, because it went to the next page, so a reader who has followed the list turns over to find out what step four is. I wrote that document, I read the proof, I sold it, and I never saw it. The tool did, on the first run it was ever allowed.

Of those 392 pages, 328 carry the warning from the top of this piece. Eighty-four percent. The smoke detector is finally hanging in a kitchen, and it is going off in five rooms out of six.

I had a reading of that ready, and I typed it into a draft: the line sits only 8.6 points below the ceiling, so of course it fires on everything, and the rule is therefore close to meaningless. One of the readers asked a question I could not answer from the text — my own piece says 40 of these pages carry a chart, so are the chart pages the ones being flagged? I had not looked. It took four minutes to look, and the answer went the other way in both directions.

Of the 392 printed pages, 239 come out of documents that carry a chart, and 182 of those are flagged: 76 percent. The other 153 pages come out of documents without one, and 146 of those are flagged: 95 percent. The chart pages are the ones getting away with it. And the flagged pages are not crowded against the line at all — the median sits at 42.5 percent, and only 75 of the 328 fall in the last ten points below the threshold. My tidy explanation was wrong twice over, and both times in the direction that flattered me. The 59-percent page at the top of this piece is not the typical case either.

Here is what the numbers actually say. The rule adds up the height of the text blocks on a page. A page built from padded cards, a callout box and a form has plenty of design on it and very little of that. My pages genuinely do carry a small amount of text per page, and the rule is measuring that correctly. What nobody has ever checked is whether the quantity it measures has anything to do with the thing a reader would call a defect. That is the entire content of the word uncalibrated, and it took a run over my own product to make me feel it rather than concede it.

In the same week I took two rules out of what I ship, because they needed to see painted pixels and nothing in the tool can see paint by itself. Eleven rules with a number I picked and two with a structural limit are what is left. Only the two structural ones may fail a build, and that buys less than it sounds like: they are free of my arbitrary line and not free of my measuring apparatus, and that apparatus was 18 pixels out three weeks ago. What holds them up is the gate that puts a stranger's document into every build, not the shape of their thresholds. The other eleven only ever warn. When one fires, open the page and look at it; if you disagree, move the threshold, because it was never right for anyone in particular. One of those eleven fires on five pages in six, which for any sane user means it is switched off. What the other ten do across 392 real pages, I have not measured — and not having measured it is the same hole one floor up. I ship them anyway, because a measurement you can ignore is still more than no measurement, and because the alternative is to hide numbers that are real even where the line is arbitrary.

That number is worth more than the zero I was going to end on. The zero said my tool had never touched reality. This one says it has, and that the first thing reality did was correct me twice in one afternoon.

There is a fair objection here, and it is not the one about downloading documents: test files are supposed to be artificial, and a chart in most of mine would have been noise. Right. The mistake was never that I invented them. It was that nothing stood next to them — twenty-one invented documents and, for a month, zero runs over anything else. Not a shortage of test files. A column I never filled in.

One last thing, which I noticed too late to design around. Four readers checked this essay before you did, and not one of them could verify a single number in it. They caught real things: two of the sharpest turns above exist because a reader asked a question I could not answer from my own draft. But I chose them, I briefed them, and I handed them material they had no way to check. The strongest evidence in this piece is the part I got wrong twice in one afternoon. Everything else is a man marking his own paper in public and hoping it counts as a control.

So be stricter with the imperative than I have been with myself. Those 74 pages are mine; they were written to be sold rather than to pass, which is the distinction that matters, and it is still a weaker one than genuinely foreign material. Genuinely foreign, I have two documents. Both of them broke the checker before they got near a rule.

Count your fixtures. Then count the runs over material you did not write for the checker — and be honest about which of those two numbers you are actually reporting.

I sell a written-up version of the rules I work by, here. Nothing above is evidence that it works.