Practonaut 0.2.0 · Experiment with budget = reference length · 16 Oct 2026 Five tasks from the bundled library, seed 1, 3 repeats each, real Excalidraw (--mirror required), --checks 0 (no check button), --max-actions = length of the reference solution (decision, restyle-flow: 6; insert-between, reorder, two-frames: 4); the reference solution is minimal for decision and two-frames, the other three tasks left slack (“proven minimal” in the outputs). Agent: the claude-code preset (Claude Code 2.1.301) through a subscription; model per the client claude-opus-5-5 or claude-haiku-5-5. 30 agent runs, no infrastructure error. Hypothesis and expectation were recorded before the first run. Fixed in advance, and result: Haiku 1 to 6 failed runs of 15 (result 6); Opus 0 to 2, and no more than Haiku (result 0); in at least one failure the agent reports “done” (result: in none, did not occur); compare per group “not distinguishable” with 0 or 1 instance regression (result: 1 each). Expected mechanism: the budget used up after a refused or superfluous call (occurred). Below are the unchanged outputs of “practonaut report --verify” for each report and of “practonaut compare --verify --fail-on-regression” for each group. The bench folders with the agents' transcripts are not published. === report --verify: haiku-ref6 === Practonaut bench claude-code 3/6 runs passed · 1/2 instances passed (at least half of all their runs) · 95 % interval 9–91 % over 2 instances · re-verified from the artifacts · real Excalidraw mirror · tool-restricted preset · model claude-haiku-5-5 (self-reported, client-attested) Label: haiku-5.5-tight World: excalidraw-library 1.1.0 (bundled library) Checks per run (--checks): 0 World content hash (SHA-256): 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da API key source (client-reported): none n = 6 runs · 1 seed · 3 repeats 2 tasks × 1 seed × 3 repeats, timeout 300 s, at most 6 act, undo or reset calls per run all 3 repeats passed on 1/2 instances mean actions 5.0 mean actions when passed 5.0 mean time per run 17.7 s efficiency vs reference length 1.20, proven minimal for 1 of 2 (> 1 = fewer actions than the reference solution) Reported by the client itself (not verified), summed over the runs that reported them: 629,117 input and 11,046 output tokens · 58 turns · cost 0.0238 USD. task runs passed actions reference length efficiency vs reference length ──────────── ──── ────── ─────── ──────────────── ────────────────────────────── decision 3 0/3 5.0 6 (minimal) — restyle-flow 3 3/3 5.0 6 1.20 Failures (3 of 6 runs) ✗ decision-s1-r1 completed Rejected actions: 2 of 7 act calls were refused by the host. the call named an outdated step or revision (1 ×) the run’s action budget was used up (1 ×) Not met: The drawing has exactly 3 arrows. (never met) Not met: One arrow from “In stock?” to “Ship”. (never met) ✗ decision-s1-r2 completed Rejected actions: 2 of 7 act calls were refused by the host. the call named an outdated step or revision (1 ×) the run’s action budget was used up (1 ×) Not met: The drawing has exactly 3 arrows. (never met) Not met: One arrow from “In stock?” to “Ship”. (never met) ✗ decision-s1-r3 completed Rejected actions: 2 of 7 act calls were refused by the host. the call named an outdated step or revision (1 ×) the run’s action budget was used up (1 ×) Not met: The drawing has exactly 3 arrows. (never met) Not met: One arrow from “In stock?” to “Ship”. (never met) Re-verified: 6 runs re-judged by the checker from their artifacts, against the world stored in the bench folder (excalidraw-library 1.1.0, content hash 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da); every verdict matches (0 not judged); 2 references re-checked. The stored world is this Practonaut's bundled library excalidraw-library 1.1.0 (same content hash). === report --verify: opus-ref6 === Practonaut bench claude-code 6/6 runs passed · 2/2 instances passed (at least half of all their runs) · 95 % interval 34–100 % over 2 instances · re-verified from the artifacts · real Excalidraw mirror · tool-restricted preset · model claude-opus-5-5 (self-reported, client-attested) Label: opus-5.5-tight World: excalidraw-library 1.1.0 (bundled library) Checks per run (--checks): 0 World content hash (SHA-256): 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da API key source (client-reported): none n = 6 runs · 1 seed · 3 repeats 2 tasks × 1 seed × 3 repeats, timeout 300 s, at most 6 act, undo or reset calls per run all 3 repeats passed on 2/2 instances mean actions 5.5 mean actions when passed 5.5 mean time per run 21.8 s efficiency vs reference length 1.10, proven minimal for 1 of 2 (> 1 = fewer actions than the reference solution) Reported by the client itself (not verified), summed over the runs that reported them: 469,392 input and 8,410 output tokens · 51 turns · cost 0.6835 USD. task runs passed actions reference length efficiency vs reference length ──────────── ──── ────── ─────── ──────────────── ────────────────────────────── decision 3 3/3 6.0 6 (minimal) 1.00 restyle-flow 3 3/3 5.0 6 1.20 Re-verified: 6 runs re-judged by the checker from their artifacts, against the world stored in the bench folder (excalidraw-library 1.1.0, content hash 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da); every verdict matches (0 not judged); 2 references re-checked. The stored world is this Practonaut's bundled library excalidraw-library 1.1.0 (same content hash). === report --verify: haiku-ref4 === Practonaut bench claude-code 6/9 runs passed · 2/3 instances passed (at least half of all their runs) · 95 % interval 20–94 % over 3 instances · re-verified from the artifacts · real Excalidraw mirror · tool-restricted preset · model claude-haiku-5-5 (self-reported, client-attested) Label: haiku-5.5-tight World: excalidraw-library 1.1.0 (bundled library) Checks per run (--checks): 0 World content hash (SHA-256): 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da API key source (client-reported): none n = 9 runs · 1 seed · 3 repeats 3 tasks × 1 seed × 3 repeats, timeout 300 s, at most 4 act, undo or reset calls per run all 3 repeats passed on 2/3 instances mean actions 3.1 mean actions when passed 3.2 mean time per run 15.4 s efficiency vs reference length 1.39, proven minimal for 1 of 3 (> 1 = fewer actions than the reference solution) Reported by the client itself (not verified), summed over the runs that reported them: 741,321 input and 13,837 output tokens · 68 turns · cost 0.0300 USD. task runs passed actions reference length efficiency vs reference length ────────────── ──── ────── ─────── ──────────────── ────────────────────────────── insert-between 3 0/3 3.0 4 — reorder 3 3/3 2.3 4 1.78 two-frames 3 3/3 4.0 4 (minimal) 1.00 Failures (3 of 9 runs) ✗ insert-between-s1-r1 completed Rejected actions: 2 of 5 act calls were refused by the host. the action did not fit the board at that moment (1 ×) the run’s action budget was used up (1 ×) Not met: The drawing has exactly 2 arrows. (never met) Not met: One arrow from “Check payment” to “Ship order”. (never met) ✗ insert-between-s1-r2 completed Rejected actions: 2 of 5 act calls were refused by the host. the action did not fit the board at that moment (1 ×) the run’s action budget was used up (1 ×) Not met: The drawing has exactly 2 arrows. (never met) Not met: One arrow from “Check payment” to “Ship order”. (never met) ✗ insert-between-s1-r3 completed Rejected actions: 2 of 5 act calls were refused by the host. the action did not fit the board at that moment (1 ×) the run’s action budget was used up (1 ×) Not met: The drawing has exactly 2 arrows. (never met) Not met: One arrow from “Check payment” to “Ship order”. (never met) Re-verified: 9 runs re-judged by the checker from their artifacts, against the world stored in the bench folder (excalidraw-library 1.1.0, content hash 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da); every verdict matches (0 not judged); 3 references re-checked. The stored world is this Practonaut's bundled library excalidraw-library 1.1.0 (same content hash). === report --verify: opus-ref4 === Practonaut bench claude-code 9/9 runs passed · 3/3 instances passed (at least half of all their runs) · 95 % interval 43–100 % over 3 instances · re-verified from the artifacts · real Excalidraw mirror · tool-restricted preset · model claude-opus-5-5 (self-reported, client-attested) Label: opus-5.5-tight World: excalidraw-library 1.1.0 (bundled library) Checks per run (--checks): 0 World content hash (SHA-256): 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da API key source (client-reported): none n = 9 runs · 1 seed · 3 repeats 3 tasks × 1 seed × 3 repeats, timeout 300 s, at most 4 act, undo or reset calls per run all 3 repeats passed on 3/3 instances mean actions 3.0 mean actions when passed 3.0 mean time per run 21.9 s efficiency vs reference length 1.44, proven minimal for 1 of 3 (> 1 = fewer actions than the reference solution) Reported by the client itself (not verified), summed over the runs that reported them: 614,982 input and 9,593 output tokens · 60 turns · cost 0.8473 USD. task runs passed actions reference length efficiency vs reference length ────────────── ──── ────── ─────── ──────────────── ────────────────────────────── insert-between 3 3/3 3.0 4 1.33 reorder 3 3/3 2.0 4 2.00 two-frames 3 3/3 4.0 4 (minimal) 1.00 Re-verified: 9 runs re-judged by the checker from their artifacts, against the world stored in the bench folder (excalidraw-library 1.1.0, content hash 6f46bc08e6a612235d29a0aa06eed3324952ad16897b0b042fc4e08ca80c33da); every verdict matches (0 not judged); 3 references re-checked. The stored world is this Practonaut's bundled library excalidraw-library 1.1.0 (same content hash). === compare: opus-ref6 → haiku-ref6 === Practonaut compare opus-5.5-tight (baseline) → haiku-5.5-tight (candidate) Baseline: claude-code · opus-5.5-tight · model claude-opus-5-5 (self-reported, client-attested) · real Excalidraw mirror · tool-restricted preset · re-verified from the artifacts · 2026-10-16 Candidate: claude-code · haiku-5.5-tight · model claude-haiku-5-5 (self-reported, client-attested) · real Excalidraw mirror · tool-restricted preset · re-verified from the artifacts · 2026-10-16 Verdict: rate difference not distinguishable at this n (exact McNemar p = 1.000; 1 only on the baseline, 0 only on the candidate) Baseline: 2/2 instances passed · 95 % interval 34–100 % over the 2 instances on both sides Candidate: 1/2 instances passed · 95 % interval 9–91 % over the 2 instances on both sides Instance regressions: ✗ decision · seed 1: passed 3/3 on the baseline, failed 3/3 on the candidate How this is compared: The unit is the instance (task × seed). Its repeats give one outcome per side: a pass when at least half of all its repeats passed; a run that was not judged counts as not passed. An instance regression needs three judged repeats on each side, all passed on the baseline and all failed on the candidate. A claim about rates needs an exact McNemar test on the instances the two sides disagree on, at α = 0.05. practonaut: 1 instance regression (passed in every baseline repeat, failed in every candidate repeat): decision seed 1 [instance_regression] (exit code 2) === compare: opus-ref4 → haiku-ref4 === Practonaut compare opus-5.5-tight (baseline) → haiku-5.5-tight (candidate) Baseline: claude-code · opus-5.5-tight · model claude-opus-5-5 (self-reported, client-attested) · real Excalidraw mirror · tool-restricted preset · re-verified from the artifacts · 2026-10-16 Candidate: claude-code · haiku-5.5-tight · model claude-haiku-5-5 (self-reported, client-attested) · real Excalidraw mirror · tool-restricted preset · re-verified from the artifacts · 2026-10-16 Verdict: rate difference not distinguishable at this n (exact McNemar p = 1.000; 1 only on the baseline, 0 only on the candidate) Baseline: 3/3 instances passed · 95 % interval 43–100 % over the 3 instances on both sides Candidate: 2/3 instances passed · 95 % interval 20–94 % over the 3 instances on both sides Instance regressions: ✗ insert-between · seed 1: passed 3/3 on the baseline, failed 3/3 on the candidate How this is compared: The unit is the instance (task × seed). Its repeats give one outcome per side: a pass when at least half of all its repeats passed; a run that was not judged counts as not passed. An instance regression needs three judged repeats on each side, all passed on the baseline and all failed on the candidate. A claim about rates needs an exact McNemar test on the instances the two sides disagree on, at α = 0.05. practonaut: 1 instance regression (passed in every baseline repeat, failed in every candidate repeat): insert-between seed 1 [instance_regression] (exit code 2)