Can a cheap model install and use TayHCode/agent-bench?

Newest observation per platform and model. Revisions, runtimes and operations can differ; this is not a ranking. Earlier observations remain in the full history.

Works

README claims checked out on 4 recorded setups. Untested combinations remain unknown.

Answers apply to each recorded setup, revision and operation. Earlier observations remain in Run details.

Works with my model and platform?
Platform	Haiku	Luna
Linux	yes	yes
Windows	blocked: setup step	not tested
macOS	yes	yes

Linux / Haiku 5.5: yes · Oct 21, 2026 · ad01b757a056 · claude-code / 2.1.300 (Claude Code)
Recorded operation: <guest-path> run --task toy-null-total --contestant stub-solver --profile byo --out /tmp/ab-rehearsal/probe-runs-checker-50f530ca59714697a5b5ad89be7f8b30

Linux / Luna: yes · Oct 22, 2026 · ad01b757a056 · codex / codex-cli 0.163.0
Recorded operation: <guest-path> demo

Windows / Haiku 5.5: blocked: setup step · Oct 22, 2026 · ad01b757a056 · claude-code / 2.1.300 (Claude Code)
Recorded operation: partial checks

macOS / Haiku 5.5: yes · Oct 22, 2026 · ad01b757a056 · claude-code / 2.1.300 (Claude Code)
Recorded operation: <guest-path> demo

macOS / Luna: yes · Oct 22, 2026 · ad01b757a056 · codex / codex-cli 0.163.0
Recorded operation: <guest-path> demo --html <guest-path>

Does what it says

1 of 1 claims checked out

The demo exercises the pipeline with two free stand-in agents and demonstrates the known fix passes while doing nothing fails.: Real input produced the expected result: agent-bench demo
Source: They cost nothing and use no model, so they test the plumbing, not any agent.
Input: toy task, stub-solver and stub-lazy
Expected: stub-solver passes and stub-lazy fails
Command: <guest-path> demo --html <guest-path>
Observed: agent-bench demo</title><style> :root{color-scheme:light dark;--bg:#fcfcfb;--card:#f4f3f0;--fg:#0b0b0b;--muted:#52514e;--line:#e2e1dc;--success:#2a78d6;--fail:#eb6834;--bar:#2a78d6;--ring:#fcfcfb} @media(prefers-color-scheme:dark){:root{--bg:#1a1a19;--card:#242423;--fg:#ffffff;--muted:#c3c2b7;--line:#383835;--success:#3987e5;--fail:#d95926;--bar:#3987e5;--ring:#1a1a19}} *{box-sizing:border-box} body{margin:0;background:var(--bg);color:var(--fg);font:16px/1.55 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif} main{max-width:980px;margin:0 auto;padding:20px 16px 64px} h1{font-size:1.7rem;lin

MCP

Local or hosted

local

Local: runs on your machine

Needs keys or browser

None seen

Maintained

Maintained

actively maintained: last commit 6 days ago; 5 commits and 1 merged PRs in 90 days; 0 open issues, 0 open PRs; no releases

Last commit Oct 15, 2026

Setup time

47 s

47 s to a working result

Issue

none found in recorded check

Examples

1 of 1 ran as expected

README quickstart demo: command completed successfully
Source: README.md / Quickstart: three commands, no login, no cost
Input: demo --html fresh file
Expected: stub-solver passes, stub-lazy fails and HTML report is written
Command: <guest-path> demo --html <guest-path>
Observed: agent runs as your own user, so a determined agent could look for it. Runs whose transcript shows that (the <i>peeked</i> flag) are left out of every table.</li><li><b>Dollars are dollar-equivalents.</b> Tokens × the dated public list prices (2026-10-09); a subscription is not billed per token but uses plan limits. It is not an invoice.</li><li><b>Privacy.</b> Contestant CLIs send the files they read to their vendors. The runner has no telemetry or uploads. Explicit task preparation downloads pinned public sources; task setup and checks may download dependencies.</li></ul></main></body></html>

Evidence

README behavior checked

Agent interfaces

CLI

Observed interfaces were exercised by the checker. Documentation alone does not confirm compatibility.

Tested with

Codex / Luna

macOS: Codex / Luna · agent codex-cli 0.163.0 · Checked operation passed · Oct 22, 2026 on ad01b757a056

Each entry describes its recorded operation and source revision. Other agent clients were not tested.

Safety

No sign of malicious behavior found in these checks

No problems traced to this repository code in these checks; a short run cannot prove safety.

Full event, path and process lists for each observed setup:

Haiku 5.5 (claude-haiku-5-5)

Checked operation passed.

Operation: <guest-path> run --task toy-null-total --contestant stub-solver --profile byo --out /tmp/ab-rehearsal/probe-runs-checker-50f530ca59714697a5b5ad89be7f8b30

Recorded scope: product operation

Agent-reported scope: Install agentbench from the checkout and run the free stand-in benchmark: `agentbench demo`, plus a from-commit toy task built and run with the stub-solver contestant.

run: TayHCode--agent-bench-claude-haiku-5-5-1792640189152867590
date: Oct 21, 2026
source commit: ad01b757a0565c35fa590fe531964019363e70d5
test version: readme-1924d8f529c6fc4500abb29916f5c9ac942027957e7b82d725d5610040cd3b4a
purpose: Initial run
platform: linux
reasoning / service tier: low / not recorded
recorded resources: 4096 MiB RAM, 2 CPUs, 32768 MiB disk
model / version: claude-haiku-5-5 / claude-haiku-5-5
agent runtime / version: claude-code / 2.1.300 (Claude Code)
checker: PASS · cli
Recorded source: https://github.com/TayHCode/agent-bench
Backend: hyperv
Tool transport: mcp-assigned-vm
Requested source pin (not an observed revision): {"commit": "ad01b757a0565c35fa590fe531964019363e70d5", "url": "https://github.com/TayHCode/agent-bench"}
time to checked operation: 720.2s
total attempt: 758.8s
reported cost (usage complete): $0.017587

run JSON · checker evidence · agent report + check · transcript excerpt · error excerpt unavailable

Checked operation passed.

Operation: <guest-path> demo

Recorded scope: product operation

Agent-reported scope: Install agent-bench (agentbench CLI) from the pinned checkout and run the free offline demo, which builds a toy repo and scores two stand-in agents; also run the documented from-commit rehearsal (toy task build, then stub-solver run reporting passed=True).

run: TayHCode--agent-bench-claude-haiku-5-5-1792641802162398530
date: Oct 22, 2026
source commit: ad01b757a0565c35fa590fe531964019363e70d5
test version: readme-6f8cbfa597c6d59f7ece0e29599b48d61f780e82e853b6ab300c3a8265684642
purpose: Initial run
platform: macos
reasoning / service tier: low / not recorded
recorded resources: 4096 MiB RAM, None CPUs, 32768 MiB disk
model / version: claude-haiku-5-5 / claude-haiku-5-5
agent runtime / version: claude-code / 2.1.300 (Claude Code)
checker: PASS · cli
Recorded source: https://github.com/TayHCode/agent-bench
Backend: tart
Tool transport: mcp-assigned-vm
Requested source pin (not an observed revision): {"commit": "ad01b757a0565c35fa590fe531964019363e70d5", "url": "https://github.com/TayHCode/agent-bench"}
time to checked operation: 418.0s
total attempt: 453.5s
reported cost (usage complete): $0.019631

run JSON · checker evidence · agent report + check · transcript excerpt · error excerpt unavailable

The recorded outcome is unverified; see the linked evidence.

Operation: partial checks

Agent-reported scope: Follow the agent-bench README quickstart (python -m pip install ., agentbench doctor, agentbench demo) and run it

run: TayHCode--agent-bench-claude-haiku-5-5-1792656967238797283
date: Oct 22, 2026
source commit: ad01b757a0565c35fa590fe531964019363e70d5
test version: readme-1924d8f529c6fc4500abb29916f5c9ac942027957e7b82d725d5610040cd3b4a
purpose: Initial run
platform: windows
reasoning / service tier: low / not recorded
recorded resources: 4096 MiB RAM, 2 CPUs, 32768 MiB disk
model / version: claude-haiku-5-5 / claude-haiku-5-5
agent runtime / version: claude-code / 2.1.300 (Claude Code)
checker: PASS
Recorded source: https://github.com/TayHCode/agent-bench
Backend: hyperv
Tool transport: mcp-assigned-vm
Requested source pin (not an observed revision): {"commit": "ad01b757a0565c35fa590fe531964019363e70d5", "url": "https://github.com/TayHCode/agent-bench"}
time to working: Not established by this run.
total attempt: 341.9s
reported cost (usage complete): $0.030976

run JSON · checker evidence · agent report + check · transcript excerpt · error excerpt unavailable

Luna (gpt-6-luna)

Checked operation passed.

Operation: <guest-path> demo

Recorded scope: product operation

Agent-reported scope: agentbench demo

run: TayHCode--agent-bench-gpt-6-luna-1792682175058527795
date: Oct 22, 2026
source commit: ad01b757a0565c35fa590fe531964019363e70d5
test version: readme-1002bc876f0002f622e14f9c5b99f9f66721fccc0356f4d53ed24f2f79a3ac5d
purpose: Initial run
platform: linux
reasoning / service tier: low / standard
recorded resources: 4096 MiB RAM, 2 CPUs, 32768 MiB disk
model / version: gpt-6-luna / gpt-6-luna
agent runtime / version: codex / codex-cli 0.163.0
checker: PASS · cli
Recorded source: https://github.com/TayHCode/agent-bench
Backend: hyperv
Tool transport: mcp-assigned-vm
Requested source pin (not an observed revision): {"commit": "ad01b757a0565c35fa590fe531964019363e70d5", "url": "https://github.com/TayHCode/agent-bench"}
time to checked operation: 139.2s
total attempt: 164.7s
reported cost: Not reported; not assumed to be zero.

run JSON · checker evidence · agent report + check · transcript excerpt · error excerpt unavailable

Checked operation passed.

Operation: <guest-path> demo --html <guest-path>

Recorded scope: product operation

Agent-reported scope: Run the documented agentbench demo

run: TayHCode--agent-bench-gpt-6-luna-1792682736439459502
date: Oct 22, 2026
source commit: ad01b757a0565c35fa590fe531964019363e70d5
test version: readme-e49b83e18e92427844d11f3118bec9288175c1147713a0bc6037ce8a852b3998
purpose: Initial run
platform: macos
reasoning / service tier: low / standard
recorded resources: 4096 MiB RAM, None CPUs, 32768 MiB disk
model / version: gpt-6-luna / gpt-6-luna
agent runtime / version: codex / codex-cli 0.163.0
checker: PASS · cli
Recorded source: https://github.com/TayHCode/agent-bench
Backend: tart
Tool transport: mcp-assigned-vm
Requested source pin (not an observed revision): {"commit": "ad01b757a0565c35fa590fe531964019363e70d5", "url": "https://github.com/TayHCode/agent-bench"}
time to checked operation: 47.0s
total attempt: 88.3s
reported cost: Not reported; not assumed to be zero.

run JSON · checker evidence · agent report + check · transcript excerpt · error excerpt unavailable

Works

Haiku 5.5 (claude-haiku-5-5)

Luna (gpt-6-luna)

More Coldpass questions