VIDA-NYU/data-gatherer

Start the data-gatherer-mcp stdio MCP server and list its tools (tools/list returns process_url_mcp and process_articles_mcp).

cd <host-path> && git log -1 --format=%H 2>/dev/null; ls -la; cat README.md 2>/dev/null | head -300
echo CDP=$CDP_URL $<env-var>; python3 --version; which node npm uv pip3 2>&1; ls <host-path> 2>&1
cd <host-path> && cat README.rst; cat requirements.txt; sed -n 1,80p pyproject.toml; ls .github/workflows
cd <host-path> && cat .github/workflows/tests.yml | head -80; ls docs; grep -ril "mcp" docs data_gatherer --include=*.rst --include=*.md --include=*.py | head -20
cd <host-path> && cat docs/installation.rst; sed -n 1,80p data_gatherer/mcp.py
cd <host-path> && python3 -m venv <host-path> && <host-path> install -q --upgrade pip 2>&1 | tail -3; export NODE_USE_ENV_PROXY=1; timeout 1500 <host-path> install -e . > /tmp/dg-install.log 2>&1; echo rc=$?; tail -5 /tmp/dg-install.log
cd <host-path> && (sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3.12-venv >/tmp/apt.log 2>&1); echo apt=$?; rm -rf <host-path>; python3 -m venv <host-path> && <host-path> install -q --upgrade pip 2>&1 | tail -3; echo ok
cd <host-path> && export NODE_USE_ENV_PROXY=1; timeout 1700 <host-path> install -e . > /tmp/dg-install.log 2>&1; echo rc=$?; tail -5 /tmp/dg-install.log; ls <host-path> | grep -i gatherer
cd /tmp && <host-path> -c "import mcp.server.fastmcp" 2>&1 | tail -2; <host-path> - <<'EOF' 2>&1 | tail -30
import json, subprocess, sys, time
p = subprocess.Popen(["<host-path>"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, cwd="/tmp")
def send(m):
    p.stdin.write(json.dumps(m)+"\n"); p.stdin.flush()
send({"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}})
print(p.stdout.readline()[:300])
send({"jsonrpc":"2.0","method":"notifications/initialized"})
send({"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}})
print(p.stdout.readline()[:1500])
p.kill()
print("stderr:", p.stderr.read()[-800:])
EOF
cd /tmp && <host-path> install -q mcp 2>&1 | tail -2; <host-path> - <<'EOF' 2>&1 | tail -30
import json, subprocess, sys, time
p = subprocess.Popen(["<host-path>"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, cwd="/tmp")
def send(m):
    p.stdin.write(json.dumps(m)+"\n"); p.stdin.flush()
send({"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}})
print(p.stdout.readline()[:300])
send({"jsonrpc":"2.0","method":"notifications/initialized"})
send({"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}})
print(p.stdout.readline()[:1500])
p.kill()
print("stderr:", p.stderr.read()[-800:])
EOF
cd /tmp && <host-path> install -q "mcp<2" 2>&1 | tail -2; <host-path> - <<'EOF' 2>&1 | tail -30
import json, subprocess, sys, time
p = subprocess.Popen(["<host-path>"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, cwd="/tmp")
def send(m):
    p.stdin.write(json.dumps(m)+"\n"); p.stdin.flush()
send({"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}})
print(p.stdout.readline()[:300])
send({"jsonrpc":"2.0","method":"notifications/initialized"})
send({"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}})
print(p.stdout.readline()[:1500])
p.kill()
print("stderr:", p.stderr.read()[-800:])
EOF
cd /tmp && <host-path> - <<'EOF' 2>&1 | tail -30
import json, subprocess
p = subprocess.Popen(["<host-path>"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, cwd="/tmp")
def send(m):
    p.stdin.write(json.dumps(m)+"\n"); p.stdin.flush()
def recv_id(i):
    while True:
        line = p.stdout.readline()
        if not line: return None
        msg = json.loads(line)
        if msg.get("id") == i: return msg
send({"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}})
recv_id(1)
send({"jsonrpc":"2.0","method":"notifications/initialized"})
send({"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}})
r = recv_id(2)
print([t["name"] for t in r["result"]["tools"]])
p.kill()
EOF
cd /tmp && <host-path> - <<'EOF' 2>&1 | tail -30
import json, subprocess
p = subprocess.Popen(["<host-path>"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, cwd="/tmp")
def send(m):
    p.stdin.write(json.dumps(m)+"\n"); p.stdin.flush()
def recv_id(i):
    while True:
        line = p.stdout.readline()
        if not line: return None
        if not line.lstrip().startswith("{"): continue
        msg = json.loads(line)
        if msg.get("id") == i: return msg
send({"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}})
recv_id(1)
send({"jsonrpc":"2.0","method":"notifications/initialized"})
send({"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}})
r = recv_id(2)
print([t["name"] for t in r["result"]["tools"]])
p.kill()
EOF
cd <host-path> && mkdir -p .coldpass && ls -la .coldpass && cat > .coldpass/probe.json <<'EOF'
{"kind":"mcp","argv":["<host-path>","env":{"NODE_USE_ENV_PROXY":"1"}}
EOF
cat > .coldpass/report.json <<'EOF'
{"outcome":"working","operation":"Start the data-gatherer-mcp stdio MCP server and list its tools (tools/list returns process_url_mcp and process_articles_mcp).","operation_scope":"Only server start and tool listing were exercised. No article extraction was run: process_url_mcp needs an LLM API key (default extraction model gpt-4o-mini, Portkey enabled by default) and network access to publisher sites, neither of which was provided. The shared CDP browser endpoint was not used.","steps":["Read README.rst (no install or run section; points to docs/installation.rst: pip install git+https://github.com/VIDA-NYU/data-gatherer.git)","python3 -m venv <host-path> (after apt-get install python3.12-venv)","<host-path> install -e . (from .github/workflows/tests.yml; succeeded)","<host-path> install 'mcp<2' (see deviations)","Started <host-path> over stdio, sent initialize, notifications/initialized, tools/list; got 2 tools; stopped the process"],"deviations":["README.rst has no install/run section; used CI workflow pip install -e . and the pyproject console script data-gatherer-mcp.","Guest lacked venv support: ran apt-get install python3.12-venv.","Server import failed: No module named 'mcp'. mcp is listed in requirements.txt but not in pyproject dependencies; installed mcp.","Unpinned mcp resolved to 2.x, which removed mcp.server.fastmcp (ModuleNotFoundError). Installed 'mcp<2' as the error message recommends. Source not edited.","The server writes a fitz deprecation warning to stdout, so MCP clients must skip non-JSON lines. Not fixed in source."],"stuck_step":null,"main_blocker":null,"suggested_fix":null}
EOF
<env-var>=<host-path> python3 -I <host-path> --validate
pgrep -fa data-gatherer-mcp || echo "no leftover server processes"