Open source · Apache-2.0 · MCP server · 0.2.0, 16 Oct 2026
Your agent says “done”. Practonaut checks in the real app.
Practonaut measures whether an AI agent reaches its goal in a real application, currently Excalidraw. The agent works through Practonaut’s actions over MCP, not by clicking; a pinned, real Excalidraw engine applies and confirms every action, and in the end what counts is what is on the board, not what the agent claims.
npx practonautopens the example task in your browser; the real Excalidraw after a one-time npx practonaut setup
Film · 0:33 · Instrumental music, no voice-over
What the film shows
- 0:00 claude-code · run 2 first sentence of the final message · The task is done, and `check_result` passed all 8 goal conditions on the first try. · How does it know? [Picture: the board at the end of run 2]
- 0:04 Run 2 · 16 Oct 2026 · claude-code [Picture: the starting state and the board after each of the five actions]
- 0:10 Actions logged, with who made them. [Picture: history, every visible entry “Agent · claude-code”]
- 0:13 The real Excalidraw confirms every action. [Picture: the same drawing in the embedded Excalidraw]
- 0:17 8 of 8 conditions met. [Picture: goal list, first open, then “All conditions met.”]
- 0:22 3 runs, 95 % interval, re-verified. · Excerpt of the result card · practonaut bench · 16 Oct 2026 [Picture: excerpt of the result card: “3/3 passed”, “95 % interval 43–100 % over 3 instances (one run each)”, “n = 3 runs · 1 seed · 1 repeat”]
- 0:27 Practonaut · Open source · Apache-2.0 · npx practonaut
You see every action the agent takes.
The agent practises, you watch. Here Claude Code turns a list into a styled flow: every action lands on the board, the real Excalidraw confirms it, and the history records who made it. Whether it was enough is decided only by the check at the end.
Film · 0:46
What the film shows
- 0:00 One run, nothing left out. · 16 Oct 2026 · Practonaut 0.2.0 · Task: excalidraw/restyle-flow · Agent: Claude Code 2.1.301 · Model per client: claude-opus-5-5 · Recording attempt 2 of 2 (attempt 1 also passed)
- 0:04 Agent connects [Picture: the task, then the whole player; Claude Code reads the task and the actions over MCP]
- 0:09 5 actions over MCP [Picture: the board, five actions: arrow “Begin” → “Process”, arrow “Process” → “Finish”, “Begin” becomes an ellipse, “Finish” an ellipse with green fill, “Process” 260 × 120]
- 0:24 Check [Picture: goal list, first open, then “All conditions met.”]
- 0:29 Agent's final message [Picture: history, entries 1–4 each “Agent · claude-code”; below, the Player's box for the last action: “What this changed”, “Resized step “Process” to 260 × 120.”] · claude-code, final message, first sentence: · The task is done, and `check_result` passed all 8 goal conditions on the first try. · Action 5, as the Player reports it
- 0:36 Real Excalidraw [Picture: the “Excalidraw” tab, the same drawing in the embedded Excalidraw]
- 0:40 Separate checker · Then a separate process checks the export. · $ practonaut check practonaut.excalidraw \ · --world library.pnworld --task restyle-flow \ · --seed 42 --pair practonaut-session.json · { "ok": true, "scope": "end-state-only", · "passed": true, …, · "counterprobes": { "empty": false, · "unchanged": false, "wrongArtifact": false }, … } · An empty, an unchanged and a wrong submission all fail. · run after the recording · output and paths abridged
- 1Agent · claude-codeConnected “Begin” → “Process”.
2cac5d9a - 2Agent · claude-codeConnected “Process” → “Finish”.
2fc75865 - 3Agent · claude-codeStyled step “Begin”: ellipse, black line, no fill.
9773051b - 4Agent · claude-codeStyled step “Finish”: ellipse, black line, green fill.
1e652855 - 5Agent · claude-codeResized step “Process” to 260 × 120.
df11ce54
History of this run (5 actions)
- 1Agent · claude-codeConnected “Begin” → “Process”.
2cac5d9a - 2Agent · claude-codeConnected “Process” → “Finish”.
2fc75865 - 3Agent · claude-codeStyled step “Begin”: ellipse, black line, no fill.
9773051b - 4Agent · claude-codeStyled step “Finish”: ellipse, black line, green fill.
1e652855 - 5Agent · claude-codeResized step “Process” to 260 × 120.
df11ce54
Who made each action from the player view; actions and state hashes from the export.
Film a gapless recording of a real run, task restyle-flow
How it was measured
The clock in the picture shows recording time; speed-ups (2×, 6×) are marked in the picture. At the end the separate checker process, run after the recording. Date, version, model and attempt in the opening card.
Three parts decide, and the agent is none of them.
This is how the benchmark command bench scores a run:
- Host: records every action with a state hash and who made it.
- Real Excalidraw: applies and reads it back, with the engine after one
setup. - Separate checker: its own process and parser, counter-checks first.
The agent sees what a person sees in the player: the task, its goal conditions, the check button. Reference solutions, near misses (almost-right drawings that must fail) and held-out check data never reach it through MCP.
- Agent, connected through MCP over stdio: sees the task, the goal conditions and the check button.
- 1, the Practonaut host: keeps the history with a state hash and who made each action.
- 2, Excalidraw 0.18.1, pinned and headless: applies each action and reads it back. If the result disagrees, the action is rejected and goes back to the host.
- 3, the checker in its own process: replays the history, parses the export itself and runs the counter-checks first.
- Verdict: pass or fail.
- Held back and not reachable via MCP: reference solution, near misses and check data. When scoring, the checker reads them.
setupDetails
The state cannot be written past the host, by people or agents; an action Excalidraw disagrees with is rejected. The checker replays the history with its own interpreter; an empty, an unchanged and a wrong submission must fail first. While practising, the host checks itself, after a comparison with the real Excalidraw.
From the first command to the first measurement.
Since 0.2.0 every action replies only with what changed: on excalidraw/connect 866 bytes instead of 12 to 63 KB, less context for your agent (0.2.0 CHANGELOG on GitHub ↗).
On Windows (not WSL) an MCP client needs cmd /c in front of npx. The Claude Code preset passes no provider keys: a set ANTHROPIC_API_KEY would otherwise switch Claude Code silently to API billing.
- Apache-2.0
- Node.js 22.18 or newer
- no VM, no Docker
- macOS (development)
- Linux and Windows in CI, see limits
- no telemetry, no account
- tasks CC0
# 1 · look: example task in the browser $ npx practonaut # 2 · real engine, once: 105–128 MB $ npx practonaut setup # 3 · connect an agent, here Claude Code $ claude mcp add practonaut -- npx -y practonaut mcp \ --task excalidraw/connect # 4 · measure $ npx practonaut bench --agent claude-code \ --tasks excalidraw/connect
How it was measured
setup downloads the Chromium headless shell: 105 MB (Apple silicon) to 128 MB (Linux), per the 0.2.0 CHANGELOG.
What it measures if your agent doesn’t draw.
What is measured is not the drawing but your agent: model, prompt and MCP client, today with Practonaut’s own actions. Whether a change to them helped can be checked on tasks whose result can be decided strictly. Such a task is a small file: the instruction in two languages, label variants, and next to it a start and a goal drawing (the decision task on GitHub ↗).
Excalidraw comes first: it runs as a pinned, static engine without a server or Docker, and its state can be read back after every action and checked deterministically. Whether results transfer to other applications has not been studied. Next on the roadmap are draw.io (a spike passed on 14 Oct 2026) and a mode in which your agent works with its own tools (target for 0.3.0, not a promise).
Your own application? Not yet. Today there is one adapter, Excalidraw. What a second one must do and how it is accepted is described in the adapter guide; a proposal starts there with an issue (adapter guide on GitHub ↗).
| Approach | scores | runs on |
|---|---|---|
Eval tools such as promptfoo, Inspect or Claude Code’s plugin eval | answers, tool calls and files, by rule or model judgement | local, CI, containers |
| Agent benchmarks such as OSWorld or WebArena-Verified | end state after desktop or web tasks in real programs or self-hosted websites, a check script per task | VM or Docker |
| Official MCP servers of Excalidraw and draw.io | no goal verdict: they let agents draw | — |
| Practonaut | state in the real app after semantic actions, separate checker process, re-verifiable report; a person and an agent in the same session | npx, engine after one setup, no VM or container |
- Start, your step: a drawing with three loose boxes.
- Goal, your step: the same boxes connected, one of them an ellipse.
- Read the difference: steps, labels, shapes, colours, frames, groups, arrows.
- Four probes: the goal passes, the start fails, the reference solution passes, near misses fail.
- Task written: the folder my-task with world and oracle.json.
- If one probe is wrong, nothing is written.
Two drawings make a task.
Draw the start and the goal in Excalidraw. Practonaut reads the difference and turns it into the goal conditions; the task is written only when all four probes hold.
The player does this too: under “Create” → “Draw start and goal” you draw both in the embedded editor. For one repair workflow there is also a demonstration path, where a second, separate recording confirms each rule.
$ npx practonaut tasks from-scenes \ start.excalidraw goal.excalidraw \ --out my-task Task written: 3 steps, 2 arrows, 0 frames in the goal; 7 goal conditions, 5 error classes. Checked: the goal scene passes, the start scene fails; the reference solution (2 actions) passes; 4 near misses fail. Practise it: practonaut play my-task/world
How it was measured
Commit d6798a9; start and goal drawing from the repository’s CI recipe. Lines wrapped, paths shortened.
New model, new prompt: did it help?
bench starts a fresh session for every run and has an agent solve the same tasks: Claude Code as a ready-made preset, or your own agent through a command template. The report gives pass rates as fractions with a 95 % interval, shows failed runs first and says why: which goal condition stayed open and after which action it was last lost.
compare puts two reports side by side, instance by instance; an instance is a task with a fixed starting value (seed). A regression is an instance that passed in all three repeats before and in none after; where the sample shows no difference in rates, it says “not distinguishable at this n”.
A real case: Claude Code with Opus 5.5 as the baseline and with Haiku 5.5 as the candidate, on five of the six hardest library tasks with a budget of exactly the reference length (at most as many calls as the reference solution has actions, refused calls included) and no check button. Opus passed 15 of 15 runs, Haiku 9 of 15.
$ npx practonaut compare \ opus/report.json haiku/report.json \ --verify --fail-on-regression Practonaut compare opus-5.5-tight (baseline) → haiku-5.5-tight (candidate) Verdict: rate difference not distinguishable at this n (exact McNemar p = 1.000; 1 only on the baseline, 0 only on the candidate) Baseline: 2/2 instances passed · 95 % interval 34–100 % over the 2 instances on both sides Candidate: 1/2 instances passed · 95 % interval 9–91 % over the 2 instances on both sides Instance regressions: ✗ decision · seed 1: passed 3/3 on the baseline, failed 3/3 on the candidate practonaut: 1 instance regression (passed in every baseline repeat, failed in every candidate repeat): decision seed 1 [instance_regression]
In the terminal: the group of the two tasks with reference length 6 (decision, restyle-flow), one instance each with three repeats; instances are counted: Opus 2 of 2, Haiku 1 of 2. Across all five tasks: Opus 15 of 15 runs, Haiku 9 of 15.
--checks 0 · 3 runs each EvidenceHow it was measured
Claude Code 2.1.301 through a subscription, model per the client; real Excalidraw, seed 1. Shown: the group decision and restyle-flow (at most 6 calls per run); the second group (insert-between, reorder, two-frames, at most 4) shows an instance regression on insert-between. Hypothesis and expectation recorded before the first run; 30 agent runs, every report re-verified with report --verify. Output shortened (two header lines with agent and model, explanation), paths shortened, wrapped. The output was re-run from the archive of the bench folders; sources byte-identical to v0.2.0.
The rate test says: not distinguishable at this n. The instance regression still shows exactly where it tips over.
In all six failures Haiku itself said the task was not finished; the host signals the end of the budget.
decision, excerptHow it was measured
report --html --verify, light version, freshly rendered; all three Haiku runs on decision give the same reasons. The excerpt ends below the reasons; the report then lists the goal conditions one by one.
In CI
In GitHub Actions, a recipe fails the job on a regression.
What a run costs: each of the three runs of a bench with Claude Code (model per the client: Opus 5.5) on three of the six hardest library tasks took 18 to 23 s and, as the client reported it, cost USD 0.10 to 0.12 list-price equivalent, billed through a subscription.
- name: Bench the agent (three repeats per instance) … - name: Compare with the baseline (fails on an instance regression) run: npx --yes --package "$PRACTONAUT" -- practonaut compare "$BASELINE" "$BENCH/report.json" --fail-on-regression --html "$OUT/compare.html"
examples/ci/practonaut-bench.yml, 0.2.0 on GitHub ↗How it was measured
Verbatim from the file; left-out lines as “…”, indentation reduced, long lines wrapped. The recipe needs a committed baseline with the same settings and, for Claude Code, a subscription token as a repository secret.
New task families ship only when they separate models.
For 0.2.0 we generated five task families. A family is published only once it passes a gate whose rules were committed before its test tasks existed: on 26 held-out instances in the real Excalidraw, the reference solution passes all of them, random actions at most 5 %, and every near miss fails; the strongest model solves between 10 and 80 %, the weakest at least 20 points fewer, confirmed by an exact McNemar test (p < 0.05).
| Bounds | Claude 5.5 | ||||
|---|---|---|---|---|---|
| Reference(should pass) | Random(should fail) | Opus | Sonnet | Haiku | |
| Repair | 26 | 0 | 26 | 26 | 26 |
| Injection in the drawing | 26 | 0 | 26 | 26 | 26 |
| Long horizon, look-alike labels | 26 | 0 | 26 | 26 | 26 |
| Limited checks | 26 | 0 | 26 | 26 | 26 |
| Tight action budget | 26 | 0 | 26 | 26 | 241 |
1 Haiku 5.5 solved 24 of 26 under a tight action budget.
How it was measured
Each family’s declaration committed before its held-out set; deviations recorded before the affected verdict. Test power ≈ 0.42 for a 20-point difference: “does not pass” does not mean “equal”.
Claude Opus and Sonnet 5.5 solved all 130 instances, Haiku 5.5 solved 128. No family leaves headroom, none separates the models; so none is published.
Agents and models from other vendors, local models and your own tool descriptions have not been measured. Today you measure your agent on the library or on your own tasks. Know a task your agent fails at? Propose it on GitHub.
What it does not do today.
- Not signed
report --verifyre-judges a report from its artifacts. Whoever can write the bench folder can replace those files too. - Separate checker only when scoringIt runs in
bench,checkandreport --verify, and when a task is created (tasks from-scenes, the demonstration path). While practising, the host checks itself, after a comparison with the real Excalidraw. - Measured with Claude models onlySo far measured with Claude models only; other agents run through command templates but have not been tried.
- No trainingPractonaut does not train models; it measures.
- PlatformsmacOS is the development platform. Linux and Windows run in the automated checks (CI) but have not been tried by hand; the real engine is checked automatically on Linux only.
- One applicationExcalidraw 0.18.1 is the only one so far.
- Semantic actionsAgents act through Practonaut’s actions (add a step, connect, rename …), not by clicking a screen. This measures whether an agent reaches the goal in the app’s model, not whether it can operate the GUI.
- Not sealedAgents run as your OS user. An agent with its own file or shell tools could read the task library on disk, reference solutions included; the Claude Code preset allows only Practonaut’s tools.
- Small evidenceWhether practising here makes people better at the real tool has not been studied.
- Model-only without ChromiumTasks work, but nothing is checked against the real app, and Practonaut says so on screen.
- DataPractonaut sends no data: no telemetry, no account, data stays local. While practising, its process makes no network calls. Only
npx(npm queries the registry and fetches the package) andsetup(fetches Chromium once) use the network;--require-offlineruns a short network probe at start to confirm there is none. Your agent talks to its own model provider. - AccessibilityScreen-reader and 200 % zoom checks by a person have not been done yet.
Practonaut is not affiliated with Excalidraw or Anthropic. The public roadmap names a second application and agents with their own tools for 0.3.0, target March 2027; dates are targets, not promises, and the content depends on a usage checkpoint (roadmap on GitHub ↗).
Where it comes from
Practonaut is developed and maintained by Dargel Solutions in Frankfurt am Main (contact). It is released under the Apache License 2.0, commercial use included. The Excalidraw engine and fonts keep their own licences; the task library and the example world are CC0 1.0.
Practonaut is written mostly by AI coding agents; since version 0.1.0 the commits name the model as co-author. Dargel Solutions sets the direction, is responsible for the result and publishes every release itself.
Generative AI is used to create, maintain and review this page. Dargel Solutions is responsible for publication.
In the MCP Registry, Practonaut is listed as io.github.godarg/practonaut.
