orq-ai/evaluatorq

Ran the README quick-start example (examples/lib/basics/support_agent_eval.py) with the venv Python: two agents scored by string_contains_evaluator over 3 datapoints; output shows agent-v1 0.33, agent-v2 1.00, 0 failed jobs, 'Evaluation completed successfully'. Run locally with ORQ_API_KEY unset.

cd <host-path> && git log -1 --format=%H 2>/dev/null; ls -la; cat README.md | head -200
cd <host-path> && ls -la .github/workflows 2>/dev/null; cat package.json 2>/dev/null | head -60; which node npm python3 pip3; node -v; npm -v
cd <host-path> && sed -n 200,400p README.md | grep -n -E "^#|uv run|eq |bash" | head -60; grep -n -E "uv run eq|eq (--help|version)|eq " README.md docs/*.md 2>/dev/null | head -20; ls src/evaluatorq; grep -n -A3 "\[project.scripts\]" pyproject.toml
cd <host-path> && (command -v uv || (curl -LsSf https://astral.sh/uv/install.sh | sh)) 2>&1 | tail -3; export PATH=$HOME/.local/bin:$PATH; export NODE_USE_ENV_PROXY=1; uv --version && uv sync --all-extras 2>&1 | tail -15; ls .venv/bin | grep -E "^eq$|^python$"
cd <host-path> && export PATH=$HOME/.local/bin:$PATH NODE_USE_ENV_PROXY=1; V=$(grep -m1 -oE '^## \[?[0-9]+\.[0-9]+\.[0-9]+' CHANGELOG.md | grep -oE '[0-9]+\.[0-9]+\.[0-9]+'); echo "version=$V"; export SETUPTOOLS_SCM_PRETEND_VERSION=$V; uv sync --all-extras 2>&1 | tail -8; ls .venv/bin | grep -E "^eq$"
cd <host-path> && grep -n -m3 -E "^## |version" CHANGELOG.md | head; grep -n -A4 "hatch.version" pyproject.toml
cd <host-path> && export PATH=$HOME/.local/bin:$PATH NODE_USE_ENV_PROXY=1 SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0.dev0 SETUPTOOLS_SCM_PRETEND_VERSION_FOR_EVALUATORQ=0.0.0.dev0; uv sync --all-extras 2>&1 | tail -8; ls .venv/bin | grep -E "^eq$"
cd <host-path> && unset ORQ_API_KEY; ls examples/lib/basics/support_agent_eval.py && timeout 300 <host-path> examples/lib/basics/support_agent_eval.py 2>&1 | tail -40; echo "rc=${PIPESTATUS[0]}"
cd <host-path> && mkdir -p .coldpass && cat > .coldpass/probe.json <<'EOF'
{"kind":"cli","argv":["<host-path>","expected_text":"Evaluation completed successfully"}
EOF
cat > .coldpass/report.json <<'EOF'
{"outcome":"working","operation":"Ran the README quick-start example (examples/lib/basics/support_agent_eval.py) with the venv Python: two agents scored by string_contains_evaluator over 3 datapoints; output shows agent-v1 0.33, agent-v2 1.00, 0 failed jobs, 'Evaluation completed successfully'. Run locally with ORQ_API_KEY unset.","operation_scope":"Core evaluation only. Red teaming, simulation, dashboard, and coding-agent Docker mode were not exercised; they need an ORQ_API_KEY, Docker, or an LLM provider key. Full pytest/ruff/ty suites intentionally not run.","steps":["Installed uv 0.13.1 via https://astral.sh/uv/install.sh (README 'Install' uses uv)","Set SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0.dev0 and SETUPTOOLS_SCM_PRETEND_VERSION_FOR_EVALUATORQ=0.0.0.dev0, then ran 'uv sync --all-extras' in <host-path> (creates .venv with eq CLI)","Ran examples/lib/basics/support_agent_eval.py (README Quick start) with .venv/bin/python: exit 0"],"deviations":["'uv sync --all-extras' (README Install / uv add) failed first with hatch-vcs 'setuptools-scm was unable to detect version' because the checkout has no .git directory. Worked around by setting SETUPTOOLS_SCM_PRETEND_VERSION env vars (no source edits). The version 0.0.0.dev0 is a placeholder."],"stuck_step":null,"main_blocker":null,"suggested_fix":null}
EOF
<env-var>=<host-path> python3 -I <host-path> --validate