Haiku 5.5
Native agent · Needs credentials · full use untestedOperation / attempted scope limited checks
Agent-reported scope: Build the Brave Search MCP server from the README and run its stdio MCP server, then call a search tool (e.g. brave_web_search).
Run details
- run
- brave--brave-search-mcp-server-claude-haiku-5-5-1792808070937972573
- date
- Oct 23, 2026
- time to working
- not recorded
- product hosting / execution
- Not recorded
- recorded operation scope
- limited checks
- model
- claude-haiku-5-5
- agent / version
- claude-code / 2.1.300 (Claude Code)
- platform
- linux
- reported cost
- $0.017235
- Recorded source
- https://github.com/brave/brave-search-mcp-server
- Backend
- hyperv
- Tool transport
- mcp-assigned-vm
- Requested source pin (not an observed revision)
- {"commit": "af08e0d48e812a3966ea2211854db5f0e8eee32d", "url": "https://github.com/brave/brave-search-mcp-server"}
- Test version
- readme-bdc1b9b3c0d2673aef87e076d14084a8123ddcf09d520c1ee2e364a37c3f4bb9
- revision
- af08e0d48e812a3966ea2211854db5f0e8eee32d
Prerequisite: full use remains untested. This is not a repo defect.
No Brave Search API key is available in this guest; the server refuses to start without one.
Double-checks failed run: brave--brave-search-mcp-server-claude-haiku-5-5-1792805811632003343.
Step: Running the server and calling a tool: node dist/index.js requires BRAVE_API_KEY (or --brave-api-key / BRAVE_API_KEY_FILE).
No Brave Search API key is available in this guest; the server refuses to start without one.
Safety: Review signals: 13 known vulnerability.
Review signals: 13 known vulnerability.
OSV reports a known vulnerability in this dependency version. 6 distinct locations. 13 findings.
OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.
A short run cannot establish that code is safe.
Connections show IP addresses; proxies can hide the final hostname.
Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.
Install / setup / connection checks
[
{
"status": "PASS",
"details": {
"working": false,
"argv": [
"npm",
"run",
"build"
],
"kind": "cli",
"returncode": 0,
"stdout": "\n> @brave/[email protected] build\n> shx rm -rf dist && tsc && shx chmod +x dist/*.js\n",
"stderr": "",
"needs_credentials": false,
"needs_browser": false,
"scope": "partial: install"
},
"check": "install",
"optional": false
},
{
"status": "PASS",
"details": {
"working": false,
"argv": [
"node",
"dist/index.js"
],
"kind": "cli",
"returncode": 1,
"stdout": "",
"stderr": "Error: A Brave API key is required via --brave-api-key, BRAVE_API_KEY, --brave-api-key-file, or BRAVE_API_KEY_FILE. You can get one at https://brave.com/search/api/.\nInvalid configuration",
"needs_credentials": true,
"needs_browser": false,
"scope": "partial: missing credentials"
},
"check": "credentials_error",
"optional": false
}
]Answers recorded for this run
- README says npm run build builds the TypeScript project: Real input produced the expected result: shx chmod +x dist/*.js
Source: - `npm run build`: Build the TypeScript project
Input: npm run build
Expected: TypeScript compiles to dist
Command: npm run build
Observed: shx chmod +x dist/*.js
- The server needs a Brave Search API key to run: No Brave Search API key is available in this guest; obtaining one would require credentials, which were not requested. Startup without a key was observed to exit 1 with 'A Brave API key is required' (see probe credentials_error check).
Source: - `BRAVE_API_KEY`: Your Brave Search API key (required unless `BRAVE_API_KEY_FILE` is set)
Observed: No Brave Search API key is available in this guest; obtaining one would require credentials, which were not requested. Startup without a key was observed to exit 1 with 'A Brave API key is required' (see probe credentials_error check).
Examples
- Run the documented test suite: command completed successfully
Source: README.md Development > Available Scripts
Input: npm test
Expected: 78 tests, 0 failures
Command: npm test
Observed: rom a file --- duration_ms: 3.037651 type: 'test' ... # Subtest: rejects empty files ok 2 - rejects empty files --- duration_ms: 2.905552 type: 'test' ... # Subtest: reports missing files with the path and reason ok 3 - reports missing files with the path and reason --- duration_ms: 0.247068 type: 'test' ... 1..3 ok 25 - readBraveApiKeyFromFile --- duration_ms: 6.289499 type: 'suite' ... 1..25 # tests 78 # suites 26 # pass 78 # fail 0 # cancelled 0 # skipped 0 # todo 0 # duration_ms 25363.950692
- Local HTTP serving needs an API key; the documented serve:http script was not run because it requires BRAVE_API_KEY: Brave Search API key is required by the server (BRAVE_API_KEY) and none was available; the documented npm run serve:http and MCP Inspector steps were not run. Startup without a key was checked instead (probe credentials_error).
Source: README.md Development > Testing via MCP Inspector
Expected: documented command completes successfully
Observed: Brave Search API key is required by the server (BRAVE_API_KEY) and none was available; the documented npm run serve:http and MCP Inspector steps were not run. Startup without a key was checked instead (probe credentials_error).
Local or hosted: both
Suggested next step (agent-reported): Re-run with a Brave Search API key supplied as BRAVE_API_KEY (or BRAVE_API_KEY_FILE) in the guest, then run probe/claims for a real brave_web_search call. No change to the repository is needed.
What the agent did
Build the Brave Search MCP server from the README and run its stdio MCP server, then call a search tool (e.g. brave_web_search).
cd <guest-path> && pwd && ls -la && id -u && cat /etc/os-release | head -3 && python3 --version && (git -C . rev-parse 2>&1 | head -1) && ls /inputs 2>&1 | head; ls <guest-path> 2>&1
cd <guest-path> && cat README.md
cd <guest-path> && (node --version; npm --version) 2>&1; ls -la .coldpass secrets; cat package.json; cat server.json | head -40; echo ---; cat <guest-path> | head -80
cd <guest-path> && git init -q && git remote add origin https://github.com/brave/brave-search-mcp-server && git fetch --depth 1 origin af08e0d48e812a3966ea2211854db5f0e8eee32d 2>&1 | tail -3; git reset --mixed FETCH_HEAD 2>&1; git log --oneline -1; git status --short | head -20
cd /tmp && export NODE_USE_ENV_PROXY=1; (curl -fsSL -o /tmp/node22.tar.xz https://nodejs.org/dist/latest-v22.x/node-v22.22.0-linux-x64.tar.xz || curl -fsSL -o /tmp/node22.tar.xz https://nodejs.org/dist/latest-v22.x/SHASUMS256.txt) 2>&1 | tail -3; ls -la /tmp/node22.tar.xz 2>&1; curl -fsSL https://nodejs.org/dist/latest-v22.x/SHASUMS256.txt 2>&1 | grep linux-x64.tar.xz
mkdir -p /tmp/nodeinst && cd /tmp/nodeinst && export NODE_USE_ENV_PROXY=1 && curl -fsSL -o node.tar.xz https://nodejs.org/dist/latest-v22.x/node-v22.23.4-linux-x64.tar.xz && curl -fsSL -o SHASUMS256.txt https://nodejs.org/dist/latest-v22.x/SHASUMS256.txt && grep 'linux-x64.tar.xz' SHASUMS256.txt | sed 's/node-v22.23.4-linux-x64.tar.xz/node.tar.xz/' | sha256sum -c - && tar -xJf node.tar.xz && export PATH=/tmp/nodeinst/node-v22.23.4-linux-x64/bin:$PATH && node --version && npm --version
sudo -n ln -sf /tmp/nodeinst/node-v22.23.4-linux-x64/bin/node /usr/local/bin/node && sudo -n ln -sf /tmp/nodeinst/node-v22.23.4-linux-x64/bin/npm /usr/local/bin/npm && sudo -n ln -sf /tmp/nodeinst/node-v22.23.4-linux-x64/bin/npx /usr/local/bin/npx && node --version && cd <guest-path> && export NODE_USE_ENV_PROXY=1 && npm ci 2>&1 | tail -15
cd <guest-path> && npm run build 2>&1 | tail -5 && ls dist | head && npm test 2>&1 | tail -15
cd <guest-path> && env -u BRAVE_API_KEY timeout 10 node dist/index.js </dev/null; echo "rc=$?"; echo ---; env -u BRAVE_API_KEY timeout 10 node dist/index.js --help 2>&1 | head -30
cd <guest-path> && grep -n "credentials_error\|'checks'\|\"checks\"\|def main\|--validate\|--replay\|expected_file\|mcp_http" check.py | head -80
cd <guest-path> && sed -n 800,860p check.py; sed -n 1165,1200p check.py; sed -n 1228,1260p check.py
cd <guest-path> && grep -rniE "setInterval|cron|schedule|webhook|--once|--now|setTimeout" src --include=*.ts | grep -v test | head; echo "---"; ls .coldpass; grep -n "Available Scripts" -A 12 README.md | head -14
cd <guest-path> && env | grep -c BRAVE_API || true; cat > probe.json <<'EOF'
{"checks":[
{"check":"install","kind":"cli","argv":["npm","run","build"],"cwd":"<guest-path>","expected_returncode":0,"expected_text":"tsc"},
{"check":"credentials_error","kind":"cli","argv":["node","dist/index.js"],"cwd":"<guest-path>","stdin":"","expected_returncode":1,"expected_text":"A Brave API key is required"}
]}
EOF
cat > claims.json <<'EOF'
[
{"kind":"claim","claim":"The README says the test suite is run with npm test; the suite passes on the pinned commit","source":"- `npm test`: Run the test suite","input":"npm test","expected":"all tests pass, 0 failures","probe_kind":"cli","argv":["npm","test"],"cwd":"<guest-path>","expected_text":"# fail 0"},
{"kind":"claim","claim":"README says npm run build builds the TypeScript project","source":"- `npm run build`: Build the TypeScript project","input":"npm run build","expected":"TypeScript compiles to dist","probe_kind":"cli","argv":["npm","run","build"],"cwd":"<guest-path>","expected_text":"shx chmod +x dist/*.js"},
{"kind":"claim","claim":"The server needs a Brave Search API key to run","source":"- `BRAVE_API_KEY`: Your Brave Search API key (required unless `BRAVE_API_KEY_FILE` is set)","testable":false,"reason":"No Brave Search API key is available in this guest; obtaining one would require credentials, which were not requested. Startup without a key was observed to exit 1 with 'A Brave API key is required' (see probe credentials_error check)."}
]
EOF
cat > examples.json <<'EOF'
[
{"kind":"example","claim":"Run the documented test suite","source":"README.md Development > Available Scripts","input":"npm test","expected":"78 tests, 0 failures","argv":["npm","test"],"cwd":"<guest-path>","env":{},"expected_text":"# fail 0"},
{"kind":"example","claim":"Local HTTP serving needs an API key; the documented serve:http script was not run because it requires BRAVE_API_KEY","source":"README.md Development > Testing via MCP Inspector","reason":"Brave Search API key is required by the server (BRAVE_API_KEY) and none was available; the documented npm run serve:http and MCP Inspector steps were not run. Startup without a key was checked instead (probe credentials_error)."}
]
EOF
cat > cycles.json <<'EOF'
{"detected":false,"reason":"Entry points inspected: README and package.json scripts (build, test, serve:http, inspector) and src/*.ts for setInterval/cron/schedule/webhook/--once/--now. The only timer is a poll delay (setTimeout pollInterval) inside the summarizer request in src/tools/summarizer/index.ts, which is not a scheduled cycle. The server runs on demand over stdio or HTTP.","checks":[]}
EOF
python3 -I -m json.tool probe.json >/dev/null && python3 -I -m json.tool claims.json >/dev/null && python3 -I -m json.tool examples.json >/dev/null && python3 -I -m json.tool cycles.json >/dev/null && echo OK
cd <guest-path> && <env-var>=<guest-path> python3 -I <guest-path> --validate --replay 2>&1 | head -c 6000
cd <guest-path> && cat > claims.json <<'EOF'
[
{"kind":"claim","claim":"README says npm run build builds the TypeScript project","source":"- `npm run build`: Build the TypeScript project","input":"npm run build","expected":"TypeScript compiles to dist","probe_kind":"cli","argv":["npm","run","build"],"cwd":"<guest-path>","expected_text":"shx chmod +x dist/*.js"},
{"kind":"claim","claim":"The server needs a Brave Search API key to run","source":"- `BRAVE_API_KEY`: Your Brave Search API key (required unless `BRAVE_API_KEY_FILE` is set)","testable":false,"reason":"No Brave Search API key is available in this guest; obtaining one would require credentials, which were not requested. Startup without a key was observed to exit 1 with 'A Brave API key is required' (see probe credentials_error check)."}
]
EOF
python3 -I -m json.tool claims.json >/dev/null && cd <guest-path> && <env-var>=<guest-path> python3 -I <guest-path> --validate --replay > /tmp/validate1.json 2>&1; python3 -I -c "
import json,sys
d=json.load(open('/tmp/validate1.json'))
print('valid:',d['valid']); print('problems:',d['problems'])
r=d.get('replay',{})
print('replay status:',r.get('status'), 'strength:', r.get('details',{}).get('working'), r.get('evidence_strength'))
"
cd <guest-path> && cat > report.json <<'EOF'
{"outcome":"needs_credentials",
"fault_owner":"prerequisite",
"operation":"Build the Brave Search MCP server from the README and run its stdio MCP server, then call a search tool (e.g. brave_web_search).",
"operation_scope":"Pinned commit af08e0d48e812a3966ea2211854db5f0e8eee32d on Ubuntu 24.04 (node 22.23.4, npm 10.9.9). The build and unit tests were run. A real tool call was NOT made because the server requires a Brave Search API key (BRAVE_API_KEY), which is not available in this guest and was not obtained.",
"steps":[
"git init; git remote add origin https://github.com/brave/brave-search-mcp-server; git fetch --depth 1 origin af08e0d48e812a3966ea2211854db5f0e8eee32d; git reset --mixed FETCH_HEAD",
"Installed Node.js 22.23.4 (README requires Node 22.x) from nodejs.org tarball, SHA256 verified against SHASUMS256.txt; linked into /usr/local/bin",
"npm ci (runs prepare: format + build)",
"npm run build (README Local Build) - succeeded",
"npm test (README Development > Available Scripts) - 78 tests, 0 failures",
"node dist/index.js (README Command Line Options / stdio default) with no API key - exits 1 with 'A Brave API key is required via --brave-api-key, BRAVE_API_KEY, ...'"
],
"deviations":[
"node/npm were not installed in the guest; installed Node 22.23.4 from the official nodejs.org tarball instead of the README (which gives no installer). Node 22.x is the README-required version.",
"The Brave Search API key was not available, so the documented tool calls, serve:http, and MCP Inspector steps were not run. Only the missing-key startup error was checked.",
"git init and fetch of the pin were run because the snapshot had no git metadata."
],
"stuck_step":"Running the server and calling a tool: node dist/index.js requires BRAVE_API_KEY (or --brave-api-key / BRAVE_API_KEY_FILE).",
"main_blocker":"No Brave Search API key is available in this guest; the server refuses to start without one.",
"plain_failure":"The server builds and its 78 tests pass, but it cannot actually search without a Brave Search API key. That key was not provided, so no search was run and the product's real output was not observed.",
"suggested_fix":"Re-run with a Brave Search API key supplied as BRAVE_API_KEY (or BRAVE_API_KEY_FILE) in the guest, then run probe/claims for a real brave_web_search call. No change to the repository is needed."
}
EOF
python3 -I -m json.tool report.json >/dev/null && echo OK && cd <guest-path> && <env-var>=<guest-path> python3 -I <guest-path> --validate --replay 2>&1 | python3 -c "import json,sys; d=json.load(sys.stdin); print(d['valid'], d['problems'])"run JSON · checker evidence · agent report + check · transcript excerpt unavailable · error excerpt unavailable