VelvetSP/web-retrieval-mcp

No problems traced to this repository code in these checks. 2 static matches retained as context.

Reviewed as expected for a test run: attempted writes that failed, so nothing was written; resolver address lookups that send no data.

2 hardcoded secret matches retained as context; full list in the evidence.

OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.

A short run cannot establish that code is safe.

Connections show IP addresses; proxies can hide the final hostname.

Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.

Full safety evidence →

• Only http and https initial URLs are accepted; an ftp URL is refused: Real input produced the expected result: refused non-http(s) URL scheme: ftp

  Source: Only `http` and `https` initial URLs are accepted.

  Input: {"tool": "web_fetch", "arguments": {"url": "ftp://example.com/file", "render": "never"}}

  Expected: refused non-http(s) URL scheme

  Command: web-retrieval-mcp tools/call {"tool": "web_fetch", "arguments": {"url": "ftp://example.com/file", "render": "never"}}

  Observed: refused non-http(s) URL scheme: ftp", "type": "text"}], "isError": false, "structuredContent": {"result": "RETRIEVAL_FAILED: ftp://example.com/file — refused non-http(s) URL scheme: ftp"}}

• Initial hosts must resolve to globally routable addresses; localhost is refused: Real input produced the expected result: refused non-public address for 'localhost'

  Source: Initial hosts must resolve exclusively to globally routable addresses.

  Input: {"tool": "web_fetch", "arguments": {"url": "http://localhost:8/", "render": "never"}}

  Expected: refused non-public address

  Command: web-retrieval-mcp tools/call {"tool": "web_fetch", "arguments": {"url": "http://localhost:8/", "render": "never"}}

  Observed: refused non-public address for 'localhost': ::1", "type": "text"}], "isError": false, "structuredContent": {"result": "RETRIEVAL_FAILED: http://localhost:8/ — refused non-public address for 'localhost': ::1"}}

• Tools resolve credentials lazily and report what a selected route is missing: Real input produced the expected result: no usable credential for EXA_API_KEY

  Source: Tools resolve credentials lazily

  Input: {"tool": "web_search", "arguments": {"query": "python package metadata"}}

  Expected: search reports the missing EXA_API_KEY credential

  Command: web-retrieval-mcp tools/call {"tool": "web_search", "arguments": {"query": "python package metadata"}}

  Observed: no usable credential for EXA_API_KEY; set EXA_API_KEY, add it to <guest-path>, or configure the optional keyring/OS secret store | firecrawl: no usable credential for FIRECRAWL_API_KEY; set FIRECRAWL_API_KEY, add it to <guest-path>, or configure the optional keyring/OS secret store", "type": "text"}], "isError": false, "structuredContent": {"result": "SEARCH_FAILED: python package metadata — exa: no usable credential for EXA_API_KEY; set EXA_API_KEY, add it to <guest-path>, or configure

Examples

• Offline SSRF guard tests pass against the installed source tree: command completed successfully

  Source: test_ssrf_guard.py

  Input: no arguments

  Expected: ALL TESTS PASS

  Command: <guest-path> test_ssrf_guard.py

  Observed: ['<resolve-error>'] want=['<resolve-error>'] OK flush: forbidden host -> blocked url: got=['http://127.0.0.1/x'] want=['http://127.0.0.1/x'] OK flush: public host not blocked: got=False want=False OK flush: 127.0.0.1 cached False: got=False want=False OK flush: 8.8.8.8 cached True: got=True want=True OK flush: drains pending: got=[] want=[] == network-layer residual disclosure == OK observer docstring still discloses the issued-before-teardown residual: got=True want=True OK web_fetch docstring still discloses the Camoufox SSRF residual: got=True want=True ALL TESTS PASS

• Release gate ./run-tests.sh runs all suites and package build: Not run: the gate requires pip install -e '.[all,dev]' (the all extra runs camoufox fetch, which downloads a browser that is not permitted here) and provisions a Valkey fixture via tests/fixtures/valkey/install.sh. Only the offline test_ssrf_guard.py was run instead.

  Source: README.md Install from source / For development

  Expected: documented command completes successfully

  Observed: Not run: the gate requires pip install -e '.[all,dev]' (the all extra runs camoufox fetch, which downloads a browser that is not permitted here) and provisions a Valkey fixture via tests/fixtures/valkey/install.sh. Only the offline test_ssrf_guard.py was run instead.

• Live SSRF redirect check with a browser: Needs a local browser (render extra / Camoufox), which was not installed; the shared CDP endpoint is not wired to this product's render path.

  Source: README.md Install from source

  Expected: documented command completes successfully

  Observed: Needs a local browser (render extra / Camoufox), which was not installed; the shared CDP endpoint is not wired to this product's render path.

Local or hosted: local

Run the web-retrieval-mcp stdio MCP server and call its tools (web_fetch / web_search)

cd <guest-path> && pwd && ls -la && id -u && python3 --version && cat README.md 2>/dev/null | head -c 6000
echo "CDP:"; curl -s -m 10 -w "\nHTTP %{http_code}\n" "$CDP_URL/json/version"; echo; cat /etc/os-release | head -3
cd <guest-path> && sed -n '/## Quick start/,/## Six/p' README.md | head -c 7000; echo; echo ---; grep -n "^## \|^### " README.md | head -80
cd <guest-path> && sed -n '215,236p;326,344p;362,379p;402,425p' README.md
cd /tmp && (python3 -m venv --help >/dev/null 2>&1 && echo venv_ok || (sudo -n apt-get -o DPkg::Lock::Timeout=300 install -y python3-venv >/tmp/venv_install.log 2>&1; echo "apt rc=$?"; tail -3 /tmp/venv_install.log)); python3 -m venv <guest-path> && <guest-path> --version && <guest-path> -m pip --version
cd <guest-path> && <guest-path> -m pip install --quiet '<guest-path>,cache,keyring]' > <guest-path> 2>&1; echo rc=$?; tail -5 <guest-path>; ls <guest-path> | grep -i web
cd <guest-path> && sed -n '426,441p' README.md; echo ----; sed -n '286,300p' README.md; echo ----; env | grep -E "EXA|FIRECRAWL|TAVILY|WEB_" | sed 's/=.*/=<set>/' ; echo "env check done"
cd <guest-path> && ls src/web_retrieval_mcp | head -50; grep -rn -i "globally\|not globally\|loopback\|private address" src/web_retrieval_mcp/*.py | head -15; ls <guest-path> /inputs 2>&1 | head
cd <guest-path> && sed -n 285,330p src/web_retrieval_mcp/server.py; grep -n "def web_fetch\<email>\|def web_search" src/web_retrieval_mcp/server.py | head
mkdir -p <guest-path> && cat > <guest-path> <<'EOF'
import asyncio, sys, json
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def main():
    params = StdioServerParameters(command="<guest-path>", args=[], cwd="<guest-path>")
    async with stdio_client(params) as (r, w):
        async with ClientSession(r, w) as s:
            await s.initialize()
            tools = await s.list_tools()
            print("TOOLS:", [t.name for t in tools.tools])
            res = await s.call_tool(sys.argv[1], json.loads(sys.argv[2]))
            for c in res.content:
                print("RESULT:", getattr(c, "text", c))
            print("isError:", res.isError)
asyncio.run(main())
EOF
cd <guest-path> && timeout 120 <guest-path> -I <guest-path> web_fetch '{"url":"http://127.0.0.1:9/","render":"never"}' 2>&1 | tail -30
cd <guest-path> && sed -i 's/res.isError/res.is_error/' <guest-path> && timeout 120 <guest-path> -I <guest-path> web_fetch '{"url":"http://127.0.0.1:9/","render":"never"}' 2>&1 | tail -30
cd <guest-path> && timeout 120 <guest-path> -I <guest-path> web_search '{"query":"python package metadata"}' 2>&1 | tail -8; timeout 120 <guest-path> -I <guest-path> web_fetch '{"url":"http://localhost:8/","render":"never"}' 2>&1 | tail -3
cd <guest-path> && timeout 120 <guest-path> -I <guest-path> web_fetch '{"url":"ftp://example.com/file","render":"never"}' 2>&1 | tail -2; cd <guest-path> && grep -n "Only \`http\` and \`https\` initial URLs are accepted.\|Initial hosts must resolve exclusively to globally routable addresses." README.md; sed -n 1,60p <guest-path> | head -80
cd <guest-path> && grep -rn -i "cron\|schedul\|--once\|--now\|while True" src/ | head; echo ---; cat run-tests.sh | head -60
cd <guest-path> && cat > probe.json <<'EOF'
{"kind":"mcp","argv":["web-retrieval-mcp"],"cwd":"<guest-path>","env":{"PATH":"<guest-path>"},"tool":"web_fetch","arguments":{"url":"http://127.0.0.1:9/","render":"never"},"expected_text":"refused non-public address for '127.0.0.1'"}
EOF
python3 -m json.tool probe.json >/dev/null && echo probe_ok
cd <guest-path> && cat > claims.json <<'EOF'
[
{"kind":"claim","claim":"Only http and https initial URLs are accepted; an ftp URL is refused","source":"README.md","source_file":"README.md","quote":"Only `http` and `https` initial URLs are accepted.","input":"ftp://example.com/file","expected":"refused non-http(s) URL scheme","probe_kind":"mcp","argv":["web-retrieval-mcp"],"env":{"PATH":"<guest-path>"},"tool":"web_fetch","arguments":{"url":"ftp://example.com/file","render":"never"},"expected_text":"refused non-http(s) URL scheme: ftp","cwd":"<guest-path>"},
{"kind":"claim","claim":"Initial hosts must resolve to globally routable addresses; localhost is refused","source":"README.md","source_file":"README.md","quote":"Initial hosts must resolve exclusively to globally routable addresses.","input":"http://localhost:8/","expected":"refused non-public address","probe_kind":"mcp","argv":["web-retrieval-mcp"],"env":{"PATH":"<guest-path>"},"tool":"web_fetch","arguments":{"url":"http://localhost:8/","render":"never"},"expected_text":"refused non-public address for 'localhost'","cwd":"<guest-path>"},
{"kind":"claim","claim":"Tools resolve credentials lazily and report what a selected route is missing","source":"README.md","source_file":"README.md","quote":"Tools resolve credentials lazily and report what a selected route is missing.","input":"python package metadata with no provider keys","expected":"search reports the missing EXA_API_KEY credential","probe_kind":"mcp","argv":["web-retrieval-mcp"],"env":{"PATH":"<guest-path>"},"tool":"web_search","arguments":{"query":"python package metadata"},"expected_text":"no usable credential for EXA_API_KEY","cwd":"<guest-path>"}
]
EOF
python3 -m json.tool claims.json >/dev/null && echo claims_ok
cd <guest-path> && cat > claims.json <<'EOF'
[
{"kind":"claim","claim":"Only http and https initial URLs are accepted; an ftp URL is refused","source":"Only `http` and `https` initial URLs are accepted.","input":"ftp://example.com/file","expected":"refused non-http(s) URL scheme","probe_kind":"mcp","argv":["web-retrieval-mcp"],"env":{"PATH":"<guest-path>"},"tool":"web_fetch","arguments":{"url":"ftp://example.com/file","render":"never"},"expected_text":"refused non-http(s) URL scheme: ftp","cwd":"<guest-path>"},
{"kind":"claim","claim":"Initial hosts must resolve to globally routable addresses; localhost is refused","source":"Initial hosts must resolve exclusively to globally routable addresses.","input":"http://localhost:8/","expected":"refused non-public address","probe_kind":"mcp","argv":["web-retrieval-mcp"],"env":{"PATH":"<guest-path>"},"tool":"web_fetch","arguments":{"url":"http://localhost:8/","render":"never"},"expected_text":"refused non-public address for 'localhost'","cwd":"<guest-path>"},
{"kind":"claim","claim":"Tools resolve credentials lazily and report what a selected route is missing","source":"Tools resolve credentials lazily","input":"python package metadata with no provider keys","expected":"search reports the missing EXA_API_KEY credential","probe_kind":"mcp","argv":["web-retrieval-mcp"],"env":{"PATH":"<guest-path>"},"tool":"web_search","arguments":{"query":"python package metadata"},"expected_text":"no usable credential for EXA_API_KEY","cwd":"<guest-path>"}
]
EOF
python3 -m json.tool claims.json >/dev/null && echo claims_ok; cd <guest-path> && grep -c "Tools resolve credentials lazily" README.md
cd <guest-path> && PYTHONPATH=<guest-path> timeout 280 <guest-path> test_ssrf_guard.py > <guest-path> 2>&1; echo rc=$?; tail -5 <guest-path>
cd <guest-path> && cat > examples.json <<'EOF'
[
{"kind":"example","claim":"Offline SSRF guard tests pass against the installed source tree","source":"test_ssrf_guard.py","input":"no arguments","expected":"ALL  TESTS PASS","argv":["<guest-path>","test_ssrf_guard.py"],"cwd":"<guest-path>","env":{"PYTHONPATH":"<guest-path>"},"expected_text":"ALL  TESTS PASS"},
{"kind":"example","claim":"Release gate ./run-tests.sh runs all suites and package build","source":"README.md Install from source / For development","reason":"Not run: the gate requires pip install -e '.[all,dev]' (the all extra runs camoufox fetch, which downloads a browser that is not permitted here) and provisions a Valkey fixture via tests/fixtures/valkey/install.sh. Only the offline test_ssrf_guard.py was run instead."},
{"kind":"example","claim":"Live SSRF redirect check with a browser","source":"README.md Install from source","reason":"Needs a local browser (render extra / Camoufox), which was not installed; the shared CDP endpoint is not wired to this product's render path.","deferred":true}
]
EOF
cat > cycles.json <<'EOF'
{"detected":false,"reason":"Entry points inspected (src/web_retrieval_mcp/server.py, __main__.py, install.py). The server runs as a request/response MCP stdio or Streamable HTTP service with no cron, scheduler, --once/--now flag, loop, or webhook; the only while True loop is a transport/serve loop. No product cycle to trigger.","checks":[]}
EOF
python3 -m json.tool examples.json >/dev/null && python3 -m json.tool cycles.json >/dev/null && echo ok
cd <guest-path> && cat > report.json <<'EOF'
{"outcome":"needs_credentials","fault_owner":"prerequisite","operation":"Run the web-retrieval-mcp stdio MCP server and call its tools (web_fetch / web_search)","operation_scope":"Python 3.12 venv with the base package plus tavily, cache and keyring extras; no provider API keys (EXA_API_KEY, FIRECRAWL_API_KEY, TAVILY_API_KEY) and no local Camoufox browser. Verified offline: server starts, lists six tools, web_fetch refuses non-http(s) schemes and loopback/localhost hosts (SSRF guard), web_search reports the missing EXA_API_KEY and FIRECRAWL_API_KEY credentials. Not verified: real web search results or fetching a public page, which need provider keys.","steps":["python3 -m venv <guest-path>","<guest-path> -m pip install '<guest-path>,cache,keyring]'","web-retrieval-mcp (stdio MCP server, driven with an MCP client)","web_fetch url=http://127.0.0.1:9/ -> RETRIEVAL_FAILED refused non-public address","web_search query=python package metadata -> SEARCH_FAILED no usable credential for EXA_API_KEY","PYTHONPATH=src python test_ssrf_guard.py -> ALL  TESTS PASS"],"deviations":["Did not run pip install '.[all]' or 'python -m camoufox fetch' because the all extra's camoufox fetch downloads a browser, which is not permitted; installed the non-render extras instead.","Did not run ./run-tests.sh (release gate): needs '.[all,dev]' and a Valkey fixture install; ran only test_ssrf_guard.py.","Workspace is not a git checkout, so the pinned commit c0ee5646d2c23ae27692d52f4c74387180c6cf24 could not be verified; the provided snapshot was used as is.","Did not set any provider key or obtain credentials."],"stuck_step":null,"main_blocker":"No EXA_API_KEY or FIRECRAWL_API_KEY is configured in the guest, so web_search cannot return results and public-page fetches cannot be completed.","plain_failure":null,"suggested_fix":"Provide EXA_API_KEY and FIRECRAWL_API_KEY (optional TAVILY_API_KEY) through the environment or keys.env, then re-run web_search and web_fetch on a public URL. Unrelated to the product code: the server itself starts and its guards work."}
EOF
python3 -m json.tool report.json >/dev/null && echo report_ok
cd <guest-path> && <env-var>=<guest-path> timeout 900 python3 -I <guest-path> --validate --replay > <guest-path> 2>&1; echo rc=$?; tail -c 4000 <guest-path>
cd <guest-path> && grep -o '"problems": \[[^]]*\]' validate.log | head -c 1500; echo; grep -o '"ok": [a-z]*\|"status": "[a-z_]*"\|"verdict": "[a-z_]*"' validate.log | sort | uniq -c; head -c 600 validate.log
pgrep -af "web-retrieval-mcp|client.py" || echo "no leftover processes"; ls <guest-path>