GitHub

home / vs pi

benchmarks

Wombat vs Pi, on the same model and the same tasks.

Two harnesses, identical prompts, one deterministic task suite. Wombat finishes the 18-task suite in 133.6s to Pi's 270.2s, passes every check, and does it on fewer prompt tokens. The 10-task run adds the case Pi cannot finish at all: bounded web research.

2.02× faster across the 18-task suite 133.6s vs 270.2s total wall time
18 / 18 tasks passed, both agents deterministic checks, not taste scores
41% fewer prompt tokens 244.6k vs 416.3k, lean profile
10 / 10 on the 10-task run Pi passed 9/10 and timed out on research

The short version. Wombat is faster where the harness has to hold a whole job: it plans, switches capabilities, and stops researching when the budget says stop. Pi is the leaner machine on small, single-file work — the section below shows exactly where. Both are honest results from the same suite.

The 18-task suite

Same model for both agents, the same AGENTS.md, the same bench skill, the same local MCP server, and the same Playwright install. Each task runs in a fresh workspace and is scored by a deterministic check: exact bytes, a test result, parsed JSON/XML, ffprobe, a browser token, or a pixel match.

0 75 150 225 300 Pi 1.4.0 18 / 18 tasks Pi 1.4.0 — 270.2s 270.2s Wombat · default 18 / 18 tasks Wombat · default — 168.2s 168.2s Wombat · lean 18 / 18 tasks Wombat · lean — 133.6s 133.6s
Total wall-clock time for all 18 tasks. Lower is better. The lean profile is Wombat with the minimal tool profile and orchestration off; the default profile keeps orchestration tools loaded.
metric wombat · lean pi 1.4.0 delta
Tasks passed 18 / 18 18 / 18 tie
Total wall time 133.6s 270.2s 2.02× faster
Mean time / task 7.4s 15.0s 2.02× faster
Median time / task 6.7s 8.0s 1.19× faster
Prompt tokens 244,613 416,257 41% fewer
Uncached input 130,885 131,201 tied
Output tokens 8,509 9,574 11% fewer
Tool calls 69 76 9% fewer
Model steps 82 91 10% fewer

The default profile is 168.2s and passes 18/18 as well. It carries the orchestration tools, which cost about 2.6k schema tokens per request and pulled its input to 554.7k. Lean is the profile to quote for speed; default is the profile that can run a fleet.

The 10-task run, task by task

Four simple tasks and six multi-step tasks. Both agents saw the same instructions, skill content, MCP server and Playwright install. Wall time is the agent process; the checks are excluded.

0 80 160 240 320 exact_edit exact_edit — 16.2s 16.2s exact_edit — 8.7s 8.7s bugfix_tests bugfix_tests — 9.9s 9.9s bugfix_tests — 22.0s 22.0s cli_json cli_json — 12.3s 12.3s cli_json — 18.8s 18.8s vague_fix vague_fix — 20.5s 20.5s vague_fix — 13.1s 13.1s multi_file_refactor multi_file_refactor — 19.8s 19.8s multi_file_refactor — 13.2s 13.2s git_commit git_commit — 13.8s 13.8s git_commit — 11.2s 11.2s server_process server_process — 17.7s 17.7s server_process — 12.9s 12.9s browser_playwright browser_playwright — 16.6s 16.6s browser_playwright — 14.4s 14.4s mcp_tool mcp_tool — 12.3s 12.3s mcp_tool — 10.7s 10.7s research_bounded research_bounded — 136.8s 136.8s research_bounded — 300.1s 300.1s
Wall time per task. Pi's research bar is dashed: it hit the 300s timeout without writing answer.md.
0s 100s 200s 300s 400s Pi — exact_edit: 8.7s cumulative Wombat — exact_edit: 16.2s cumulative exact_edit Pi — bugfix_tests: 30.7s cumulative Wombat — bugfix_tests: 26.1s cumulative bugfix_tests Pi — cli_json: 49.5s cumulative Wombat — cli_json: 38.4s cumulative cli_json Pi — vague_fix: 62.6s cumulative Wombat — vague_fix: 58.9s cumulative vague_fix Pi — multi_file_refactor: 75.8s cumulative Wombat — multi_file_refactor: 78.7s cumulative multi_file_refactor Pi — git_commit: 87.0s cumulative Wombat — git_commit: 92.5s cumulative git_commit Pi — server_process: 99.9s cumulative Wombat — server_process: 110.2s cumulative server_process Pi — browser_playwright: 114.3s cumulative Wombat — browser_playwright: 126.8s cumulative browser_playwright Pi — mcp_tool: 125.0s cumulative Wombat — mcp_tool: 139.1s cumulative mcp_tool Pi — research_bounded: 425.1s cumulative Wombat — research_bounded: 275.9s cumulative research_bounded 276s 425s
Cumulative wall time. The agents track each other through the ordinary tasks, then Pi spends 300s on research while Wombat finishes in 137s.
task time input tokens steps tools result
exact_edit 16.2s 8.7s 46,058 15,180 7 / 3 6 / 2 Y Y
bugfix_tests 9.9s 22.0s 32,901 33,938 5 / 6 6 / 7 Y Y
cli_json 12.3s 18.8s 32,416 32,266 5 / 6 4 / 5 Y Y
vague_fix 20.5s 13.1s 51,228 26,083 7 / 5 10 / 4 Y Y
multi_file_refactor 19.8s 13.2s 56,470 28,190 8 / 5 10 / 6 Y Y
git_commit 13.8s 11.2s 34,471 20,422 5 / 4 4 / 3 Y Y
server_process 17.7s 12.9s 48,764 26,488 6 / 5 6 / 4 Y Y
browser_playwright 16.6s 14.4s 42,722 22,139 5 / 4 6 / 4 Y Y
mcp_tool 12.3s 10.7s 31,562 19,561 5 / 4 5 / 3 Y Y
research_bounded 136.8s 300.1s 121,552 612,036 8 / 22 9 / 36 Y N
total 275.9s / 424.9s 498,144 / 836,303 61 / 64 66 / 74 10/10 · 9/10

Values are Wombat / Pi. Steps are model requests; tools are tool calls.

The research task is the whole difference

Wombat loaded the web capability, ran two searches, was stopped by the quick-mode soft budget with a clear "synthesize now", and wrote a sourced 205-line answer.md in 136.8s. Pi's default toolset has no search: it made 35 bash calls, accumulated 612k input tokens, hit the 300s timeout, and never wrote the file.

That failure is one observation from one run, but the mechanism is not subtle. A harness without a research tool cannot finish a research task by shelling out faster.

Where Pi is still leaner

Strip out the failed research task and Pi is the lighter harness on ordinary coding work. Wombat's system prompt, tool schemas and orchestration context cost tokens on every request, and that shows up on small tasks. This is the part a benchmark page usually leaves out; it is the part that tells you which agent to reach for.

nine ordinary tasks wombat pi wombat / pi
Input tokens376,592224,2671.68×
Output tokens7,3864,7081.57×
Tool calls57381.50×
Model steps53421.26×
Wall time139.1s124.9s1.11×

Pi was faster on 7 of those 9 tasks. Wombat won bugfix_tests (9.9s vs 22.0s) and cli_json (12.3s vs 18.8s). On the 18-task suite the median is 6.7s vs 8.0s in Wombat's favour, so the result depends on which suite and which profile you ask about — quote both, or quote neither.

What actually changed

The speed is not a rewrite of the model loop. It is four harness decisions, each measured before it shipped.

01

Capability profiles

Prompt telemetry showed ~71% of the first request was tool schemas: 38–39 tools on every task. The default coding profile now exposes nine tools and loads web, LSP, verify or media only when the task asks for them.

→ /docs/tools
02

A search guard

One find / -iname cost 229s in the old suite. Bash now rejects searches outside the workspace before the process starts, and the system prompt points at glob and grep.

→ /docs/tools
03

Bounded research

Quick mode gives 2 searches and 1 fetch, with a hard ceiling of 4/2. When the soft budget is spent the model is told to synthesize, not to keep looping. The job prompt went from 114.6s and 13 searches to 21.1s and 2.

→ /docs/tools
04

Optional orchestration

The fleet is powerful and expensive. Orchestration tools are attached only when the orchestrator is enabled; minimal and orchestrator: false give the lean path that wins the suite.

→ /docs/agents

Method

Everything below is fixed in eval/benchmarks/pi-vs-wombat; the raw runs, task prompts and chart scripts are in the repository. If a number here cannot be traced to a file, treat it as wrong.

Limitations

Read this before quoting a number

One run per task. There is no variance estimate. The Pi research timeout is a single observation, though the failure mode is structural, not luck.

Different models per suite. The 10-task run used a paid fallback model because the default was rate-limited that day. The 18-task suite used the default. Do not compare tokens across the two suites.

No delegation exercised. Every task recorded one agent run. This measures the single-agent path, not the worker fleet.

Checks, not quality scores. The research check requires five known company names and a remote-India mention. It does not judge the prose.

Raw data

Reproduce any chart on this page from the repository. The site reads the same numbers from src/data/benchmarks.ts; the charts are drawn in SVG here, not copied from the eval assets. 

eval/benchmarks/pi-vs-wombat/results.json              # 18-task, per task
eval/benchmarks/pi-vs-wombat/EFFICIENCY.md             # phases 0-3, suite tables
eval/benchmarks/pi-vs-wombat/efficienty-10/results.json # 10-task, per task
eval/benchmarks/pi-vs-wombat/efficienty-10/README.md    # full write-up