Haiku 5.5
Native agent · Hosted MCP connection/tools-list passedChecked operation Remote hosted MCP connection · initialize + tools/list · https://mcp.exa.ai/mcp
Tested against the hosted endpoint only. Local install is untested.
Agent-reported scope: Anonymous web search through the hosted Exa MCP server (web_search_exa tool) at https://mcp.exa.ai/mcp, as documented in README 'Connect to Exa's hosted MCP server'.
MCP server returned: exa-search-server · version 3.5.0.
MCP tools returned: web_search_exa, web_fetch_exa. Tool listing does not establish authenticated tool-call success.
Run details
- run
- exa-labs--exa-mcp-server-claude-haiku-5-5-1792794299081156372
- date
- Oct 23, 2026
- time to working
- 61.7s
- product hosting / execution
- Remote hosted MCP endpoint
- recorded operation scope
- remote hosted MCP initialize + tools/list
- model
- claude-haiku-5-5
- agent / version
- claude-code / 2.1.300 (Claude Code)
- platform
- linux
- reported cost
- $0.014211
- Recorded source
- https://github.com/exa-labs/exa-mcp-server
- Backend
- hyperv
- Tool transport
- mcp-assigned-vm
- Requested source pin (not an observed revision)
- {"commit": "0b499f095c4ca145eee07b09137641c037d58449", "url": "https://github.com/exa-labs/exa-mcp-server"}
- Test version
- readme-bdc1b9b3c0d2673aef87e076d14084a8123ddcf09d520c1ee2e364a37c3f4bb9
- revision
- 0b499f095c4ca145eee07b09137641c037d58449
Safety: Review signals: 140 known vulnerability.
Review signals: 140 known vulnerability.
OSV reports a known vulnerability in this dependency version. 29 distinct locations. 140 findings.
OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.
A short run cannot establish that code is safe.
Connections show IP addresses; proxies can hide the final hostname.
Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.
Answers recorded for this run
- The hosted MCP server offers web search anonymously: Real input produced the expected result: Model Context Protocol (MCP) is an open protocol
Source: The hosted MCP server works anonymously with rate limits.
Input: {"tool": "web_search_exa", "arguments": {"query": "Model Context Protocol specification"}}
Expected: Search results with MCP text returned by web_search_exa
Command: tools/call {"tool": "web_search_exa", "arguments": {"query": "Model Context Protocol specification"}}
Observed: Model Context Protocol (MCP) is an open protocol that enables seamless integration between LLM applications and external data sources and tools. Whether you’re building an AI-powered IDE, enhancing a chat interface, or creating custom AI workflows, MCP provides a standardized way to connect LLMs with the context they need. This specification defines the authoritative protocol requirements, based on the TypeScript schema in schema.ts. For implementation guides and examples, visit modelcontextprotocol.io. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”,
Examples
- Local tests or runnable examples for the README: README documents no local test, example or run command. Its only runnable setup is the hosted endpoint https://mcp.exa.ai/mcp (exercised by the probe and claim) and client commands (claude plugin install, codex mcp add) that need external clients. Local stdio build (npm ci, npm run build:stdio) and CI (npm run ci) are in package.json/.github and not README; node/npm is not installed in this guest, so they were not run.
Source: README.md
Expected: documented command completes successfully
Observed: README documents no local test, example or run command. Its only runnable setup is the hosted endpoint https://mcp.exa.ai/mcp (exercised by the probe and claim) and client commands (claude plugin install, codex mcp add) that need external clients. Local stdio build (npm ci, npm run build:stdio) and CI (npm run ci) are in package.json/.github and not README; node/npm is not installed in this guest, so they were not run.
Local or hosted: hosted
What the agent did
Anonymous web search through the hosted Exa MCP server (web_search_exa tool) at https://mcp.exa.ai/mcp, as documented in README 'Connect to Exa's hosted MCP server'.
cd <guest-path> && ls -la && git status 2>&1 | head -3; id -u; python3 --version; cat README.md | head -200
ls /inputs 2>&1; ls <guest-path> 2>&1; node --version 2>&1; npm --version 2>&1
cd <guest-path> && cat package.json; echo ---; cat Dockerfile; echo ---; cat env.example; echo ---; ls -la .coldpass .github/*; echo ---; cat mcp.json server.json; echo; grep -n -i "npm\|node\|run \|install" README.md | head -40
env | grep -i proxy; curl -sS -m 20 -o /dev/null -w "%{http_code}\n" https://mcp.exa.ai/mcp; curl -sS -m 20 -o /dev/null -w "%{http_code}\n" https://registry.npmjs.org/; sudo -n true && echo SUDO_OK
cd <guest-path> && git init -q && git remote add origin https://github.com/exa-labs/exa-mcp-server && git fetch --depth 1 origin 0b499f095c4ca145eee07b09137641c037d58449 2>&1 | tail -3; git reset --mixed FETCH_HEAD 2>&1 | tail -2; git log --oneline -1; git status --short | head
cd /tmp && H='-H Content-Type:application/json -H Accept:application/json,text/event-stream';
curl -sS -m 60 -D /tmp/h.txt https://mcp.exa.ai/mcp $H -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"coldpass-probe","version":"1.0"}}}' | head -c 600; echo; grep -i mcp-session /tmp/h.txt; SID=$(grep -i mcp-session-id /tmp/h.txt | awk '{print $2}' | tr -d '\r');
curl -sS -m 60 https://mcp.exa.ai/mcp $H -H "Mcp-Session-Id: $SID" -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"web_search_exa","arguments":{"query":"Model Context Protocol specification"}}}' | head -c 1200; echo
cd /tmp && H=(-H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream');
curl -sS -m 60 -D /tmp/h2.txt -o /tmp/i.txt https://mcp.exa.ai/mcp "${H[@]}" -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"coldpass-probe","version":"1.0"}}}'; SID=$(grep -i '^mcp-session-id' /tmp/h2.txt | awk '{print $2}' | tr -d '\r'); echo "SID=$SID";
curl -sS -m 60 -o /dev/null -w "notif %{http_code}\n" https://mcp.exa.ai/mcp "${H[@]}" -H "Mcp-Session-Id: $SID" -d '{"jsonrpc":"2.0","method":"notifications/initialized"}';
curl -sS -m 90 https://mcp.exa.ai/mcp "${H[@]}" -H "Mcp-Session-Id: $SID" -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"web_search_exa","arguments":{"query":"Model Context Protocol specification"}}}' -o /tmp/call.txt -w "call %{http_code}\n"; head -c 1500 /tmp/call.txt; echo
cd <guest-path> && grep -n "Connect AI agents to Exa for web search\|The hosted MCP server works anonymously with rate limits\|Connect to Exa's hosted MCP server" README.md; mkdir -p .coldpass && cat > .coldpass/probe.json <<'EOF'
{"kind":"mcp_http","url":"https://mcp.exa.ai/mcp","cwd":"<guest-path>","tool":"web_search_exa","arguments":{"query":"Model Context Protocol specification"},"expected_text":"Model Context Protocol (MCP) is an open protocol"}
EOF
cat > .coldpass/claims.json <<'EOF'
[{"kind":"claim","claim":"The hosted MCP server offers web search anonymously","source":"The hosted MCP server works anonymously with rate limits.","input":"Model Context Protocol specification","expected":"Search results with MCP text returned by web_search_exa","probe_kind":"mcp_http","url":"https://mcp.exa.ai/mcp","tool":"web_search_exa","arguments":{"query":"Model Context Protocol specification"},"expected_text":"Model Context Protocol (MCP) is an open protocol","cwd":"<guest-path>"}]
EOF
python3 -m json.tool .coldpass/probe.json >/dev/null && python3 -m json.tool .coldpass/claims.json >/dev/null && echo JSON_OK
cd <guest-path> && echo '{"detected":false,"reason":"Entry points inspected (README, package.json scripts, src/stdio-cli.ts, src/runtime-server.ts, api/): the product is an MCP server that answers tool calls on request. No cron, scheduler, loop, webhook, --once or --now trigger exists in README or source; the only schedules are in .github/workflows CI, which are not product cycles.","checks":[]}' > .coldpass/cycles.json && echo '[]' > .coldpass/examples.json && <env-var>=<guest-path> python3 -I <guest-path> --validate --replay; echo "rc=$?"
cd <guest-path> && cat > .coldpass/examples.json <<'EOF'
[{"kind":"example","claim":"Local tests or runnable examples for the README","source":"README.md","reason":"README documents no local test, example or run command. Its only runnable setup is the hosted endpoint https://mcp.exa.ai/mcp (exercised by the probe and claim) and client commands (claude plugin install, codex mcp add) that need external clients. Local stdio build (npm ci, npm run build:stdio) and CI (npm run ci) are in package.json/.github and not README; node/npm is not installed in this guest, so they were not run."}]
EOF
mkdir -p .coldpass && cat > .coldpass/report.json <<'EOF'
{"outcome":"working","fault_owner":null,"operation":"Anonymous web search through the hosted Exa MCP server (web_search_exa tool) at https://mcp.exa.ai/mcp, as documented in README 'Connect to Exa's hosted MCP server'.","operation_scope":"Remote hosted endpoint only. Requires outbound HTTPS to mcp.exa.ai; anonymous use is rate-limited. No local server was built or run; Node/npm was not installed, and the README documents no local install command. Authenticated access (API key/OAuth) and Exa Agent were not tested.","steps":["git init; git remote add origin https://github.com/exa-labs/exa-mcp-server; git fetch --depth 1 origin 0b499f095c4ca145eee07b09137641c037d58449; git reset --mixed FETCH_HEAD (checkout at 0b499f0)","Read README.md; identified hosted endpoint https://mcp.exa.ai/mcp as the only documented runnable setup","curl POST initialize to https://mcp.exa.ai/mcp (HTTP 200, serverInfo exa-search-server 3.5.0, session id issued)","curl POST notifications/initialized (HTTP 202)","curl POST tools/call web_search_exa with query 'Model Context Protocol specification' (HTTP 200, returned search results)","<env-var>=<guest-path> python3 -I <guest-path> --validate --replay: PASS"],"deviations":["/inputs/coldpass-preflight.py does not exist in this guest; used <guest-path> as the task's checker path.","Node/npm not installed (node: command not found). Not installed because the README's only runnable setup is the hosted endpoint, which needs no local runtime; the local npm build was not attempted.","Git metadata created with git init in the workspace as instructed; the workspace was not a git repo before."],"stuck_step":null,"main_blocker":null,"plain_failure":null,"suggested_fix":"None for the hosted endpoint. Optional: the README could document the local stdio build (npm ci; npm run build:stdio) and Node 20+ requirement, since env.example and package.json describe it but the README does not."}
EOF
python3 -m json.tool .coldpass/report.json >/dev/null && echo REPORT_OK; <env-var>=<guest-path> python3 -I <guest-path> --validate --replay | python3 -c 'import json,sys; d=json.load(sys.stdin); print("valid",d["valid"],"problems",d["problems"],"replay",d["replay"]["status"])'; ls -la .coldpassrun JSON · checker evidence · agent report + check · transcript excerpt unavailable · error excerpt unavailable