How to Audit a Passing Check in an AI Agent Workflow

Your agent produces a file, the checks pass, and the next run is already scheduled. Before you add another reviewer, pick one result that somebody relies on. Can the existing check tell a good version of that result from a bad one?

This article takes one workflow through that question. You will write down the expected result, trace what the check reads, introduce a known defect in a safe copy, and record what happens. The useful output is a specific finding: a defect that escaped, a failure that was caught, or a question you still cannot answer. You do not need the book to do the exercise.

A passing check can answer a smaller question

In a case covering 28 July to 2 August 2026 and described in Nobody Opens a Passing Check, an agent-operated product workflow checked spreadsheets before release. The requirement concerned keeping column headings visible. The checker used sheet length to decide where to look: it required frozen headings only on sheets with more than forty rows. Shorter sheets with unfrozen headings passed through that filter.

The historical finding was a mismatch between the requirement and the selection rule. Chapter 6, case A1 of the book describes the correction broadly: remove the length threshold, require the property on every sheet, and give reasons for legitimate exemptions. The private operating decision dated 28 July 2026 specifies the implemented rule more narrowly: examine sheets with a styled table header and at least one data row beneath it, then check that the freeze is below the header. That decision does not establish that exemption logging was implemented. This is one observation from one operation, not evidence that spreadsheet checks generally fail this way.

The distinction travels further than spreadsheets. A check can confirm that a report exists without checking its reporting period. It can validate a citation's format without reading the source. Both checks have a useful job. Trouble begins when their results are treated as evidence for a broader claim.

Keep these statements separate:

An installed tool is not a run record. And a passing run does not, on its own, establish the last statement.

Start with the reader's task

Choose a recurring workflow whose output you can inspect safely: a spreadsheet, a research brief, a support draft or an internal report. For this first exercise, choose one important property. “The agent does good work” is too broad to test.

Here is our worked example. Everything from this point that names the sample workbook or its results is a synthetic teaching example, constructed to explain the method. It is not a reconstruction of the historical file or a report of a customer run.

An agent prepares a weekly stock workbook for a colleague. The workbook has two sheets, Orders and Returns. Both contain a table with its header in row 1 and data below. The colleague needs to scroll through either table without losing the column labels.

Write the requirement before reading the checker:

In the delivered workbook, row 1 remains visible while a reader scrolls through the data on both Orders and Returns.

That sentence gives you something to observe. Open the workbook in the spreadsheet application the colleague uses, scroll each table and see whether its header stays visible. A file-level check can establish that freeze settings are present in the saved file; the application check establishes how those settings behave in the reader's view. Keep both results, and name the application you used.

If your workflow is a research brief, the equivalent sentence might concern whether its main recommendation follows from the cited sources. That needs a reader comparing the actual claim with those sources. A citation counter cannot substitute for that work. For a synthetic example, suppose a source measures processing time: the good brief reports that finding, while the bad copy adds a claim of improved accuracy with the citation unchanged. First read the source and make independent notes on what it supports. Give the good brief and source to a separate reviewer, then the bad brief and source in a separate review, using the same request: state what the source supports and list every claim in the brief it does not support. Do not reveal the change or the expected finding; compare both responses with your notes.

Trace the check to the delivered output

Follow one completed run from its input to the file somebody receives. Record the run identifier, the output's location and the check result associated with it. A filename is a start; a version identifier or a content hash, which identifies the file's exact bytes, is stronger evidence when files are overwritten.

Ask what the check actually reads. Does it open the delivered workbook, a temporary copy or a specification from which the workbook was built? Does it inspect every relevant sheet? Can missing input cause it to skip work and still finish successfully?

In our example, the old checker reads the workbook but examines only sheets with more than forty rows. Orders has sixty rows and qualifies. Returns has twenty and does not. The requirement applies to both. The checker covers one.

Write down the names as well as the count. “One sheet checked” becomes useful when you can compare it with “Orders and Returns required.” Even a matching count can conceal a substituted or repeated item. Build the expected list from the requirement and the actual output, independently of the checker's own filter. Simply lowering a row-count cutoff until the example passes would leave the selection rule dependent on length; it would not establish coverage of the required tables.

This step can already produce a finding. If you cannot establish which file the checker examined, stop calling the result evidence about the delivered file. Record that link as unknown and identify the missing record. Running more checks on another copy will not repair it.

Put one known defect through the check

Use a disposable copy or a test environment with publishing, sending and other external actions disabled. Keep the original output and its evidence intact. If you cannot isolate the check from an external action, stop here and record that limitation; the exercise does not require a live experiment.

A deliberately bad input is a negative control: you know it violates the requirement, so the relevant check should reject it. First keep a good copy and verify that it meets the chosen requirement. In our example, both sheets have their first row frozen. Run the current check on this good copy and record both its response and the isolated workflow's acceptance decision before trying the defect.

Make another copy and remove the freeze from Returns only. Leave its row count and everything else unchanged. Open this bad copy and verify that the header scrolls away. This confirms that the intended defect reached the test file; a failed edit would make the experiment meaningless.

Before running anything, write the expected response: “The checker names Returns as missing its frozen header and prevents this test output from being accepted.” Then run the same check entry point and configuration that the workflow uses, directed at the test copy. Record the actual response, including which sheets it examined. Then observe the acceptance step in that same isolated workflow: did the rejection prevent the bad copy from being accepted for delivery? A warning in a log is insufficient if the workflow proceeds. If you cannot observe that link, the end-to-end result is unknown. A timeout, unreadable file or authentication error is not a successful detection of the missing header.

The following table states the expected behaviour of the synthetic rules, not measured production results. In this example, the isolated acceptance step uses the rule result: acceptance permits the test output, while rejection or an incomplete check blocks it. A real run must demonstrate that connection.

The bad copy matters because it tests the excluded case. Removing the freeze from the long Orders sheet would show that the old checker detects some missing freezes. It would leave the short-sheet gap untouched.

After changing the check, run both good and bad copies again. The good copy tests whether valid work is accepted; the bad copy tests whether the intended defect is detected for the intended reason and blocked at acceptance. Preserve both observations. A checker that rejects everything has not solved the problem.

Prepare the missing-input case separately: direct the checker at a nonexistent file path inside the safe test folder, leaving both comparison files intact. This tests whether unavailable input is reported as incomplete rather than accepted; it does not test detection of the header defect. If your workflow reads something other than a file, name the unavailable input you test, or record this case as not run.

Turn the result into a useful finding

A completed worksheet should let someone else understand what you tested without asking you to retell the run. The download provides all fourteen fields with a filled example. Here is its finding and evidence trail. All dates, filenames, run identifiers and paths are synthetic illustrations, not supplied files or measured logs.

For your own worksheet, replace the illustrative entries with dated observations and evidence locations. If you have not performed an action, write “not run.” If you cannot tell what happened, write “unknown.” Neither means the workflow passed. The record covers this header requirement only; totals, formulas and missing orders need separate investigation.

Download the editable worksheet. It is a plain-text Markdown file: open it in a text editor, keep it locally, or copy its questions into your existing issue tracker. No account is needed.

What to do when the answer is unknown

Sometimes the most useful result is identifying why you cannot reach a conclusion. Perhaps the checker has no record of the files it opened. Perhaps you cannot construct a safe failing input. Perhaps the required outcome itself is disputed.

Name the obstacle and the next action that would remove it. “Capture the next run's input and output identities” is actionable. “Improve observability” leaves another reader to invent the work.

If a bad input is accepted, record a demonstrated gap and correct it before relying on the check for that property. If it is rejected for the right reason and the isolated acceptance step blocks it, retain both observations and the good-case result. If it is rejected but still accepted for delivery, record a separate gap at the acceptance step; the defect was detected, but the workflow did not enforce that result. If it fails for an unrelated reason, the experiment remains inconclusive. Do not count that failure as coverage.

Where the check uses a language model, one rejection establishes only what happened in that attempt. Keep the prompt, input, configuration and response; repeat the same good and bad cases to look for variation. This small exercise does not estimate a detection rate.

Separately, ask a person or a model in a fresh context to challenge the requirement and evidence without the history of building the output. Record what that reviewer actually examined. Agreement alone does not establish that the workflow enforces the result.

You now have a record of one investigated promise, the evidence behind it and a next action. Keep that record beside the workflow so the next change has something concrete to preserve.

Where the book goes further

This exercise concentrates on one way a check can miss the result it is supposed to protect. Nobody Opens a Passing Check examines other shapes of failure, including correct numbers inside false claims, reduced review scope and checks whose design leaves no meaningful route to failure. It follows the operating decisions behind those cases and includes worked templates for carrying the lessons into recurring work.

For a related software case, My Tests Were Green Because I Wrote Them follows a test suite that agreed with an incorrect assumption about an external interface. The worksheet here gives you a place to investigate the equivalent assumption in your own workflow.

Source and editorial note: The historical account distinguishes Chapter 6, case A1, in Nobody Opens a Passing Check from the more specific implementation rule in the private operating decision of 28 July 2026. The book provides the broader account and exemption advice; the narrower rule is attributed to that private decision, which is not independently inspectable through this page. The worked example and worksheet are newly created teaching material, not measured field results. This article was drafted and revised with generative AI. Model-based reviews are editorial checks, not human reader testing.