coldpass

Per-setup public labels: Linux / Haiku: Works

Static status badge · SVG · read-only JSON. Embed with ![Coldpass status](https://coldpass.dev/badge/openai--math.svg).

Works

Only older checks are saved; no current product test.

Answers apply to each recorded setup, revision and operation. Earlier observations remain in Run details.

Tested on Linux
PlatformHaikuLuna
Linuxyes

Linux / Haiku 5.5: yes · Oct 18, 2026 · 1581d4cbdd69 · claude-code / 2.1.300 (Claude Code)
Recorded operation: /usr/bin/python3 -B <host-path>

Needs keys or browser
None seen
Issue
none found in recorded check

Safety: no observations were recorded for the selected revision. Functional results do not establish a safety result.

Run details

Haiku 5.5

Native agent · Checked operation passed

Checked operation /usr/bin/python3 -B <host-path>

Agent-reported scope: Ran the default rectangular certificate checks from preprints/Complex-Matrix-Multiplication-Below-2.258-and-Rectangular-Bounds-September-24-2026/verification/README.md (python3 -B verification/certificates/checks/run_checks.py). It runs verify_rational.py, verify_construction.py and verify_baseline.py, compares outputs byte-for-byte with the saved JSON, and reports status passed (exit 0).

Run details
run
openai--math-claude-haiku-5-5-1792338576011704833
date
Oct 18, 2026
time to working
not recorded
product hosting / execution
Local command in the guest
recorded operation scope
product operation
model
claude-haiku-5-5
agent / version
claude-code / 2.1.300 (Claude Code)
platform
linux
reported cost
$0.009436
Recorded source
https://github.com/openai/math
Tool transport
mcp-assigned-vm
Test version
readme-03aea4c41d53955e3723c330e9927d2a9df627f6642624cc0078a97c73a53c8a
revision
1581d4cbdd69a0faf34b2089614fe2a814316a6a
What the agent did

Ran the default rectangular certificate checks from preprints/Complex-Matrix-Multiplication-Below-2.258-and-Rectangular-Bounds-September-24-2026/verification/README.md (python3 -B verification/certificates/checks/run_checks.py). It runs verify_rational.py, verify_construction.py and verify_baseline.py, compares outputs byte-for-byte with the saved JSON, and reports status passed (exit 0).

cd <host-path> && ls -la && cat README.md | head -200
cd <host-path> && ls -la .github/workflows 2>/dev/null; cat pyproject.toml 2>/dev/null | head -50; cat package.json 2>/dev/null | head -50; which python3 node npm uv pip3 2>&1; python3 --version
cd <host-path> && cat lean/README.md; echo ----; cat lean/ComparatorChallenges/README.md 2>/dev/null | head -80; echo ----; cat .github/workflows/lock-prs.yml; echo ----; ls lean; which lake lean elan 2>&1
cd <host-path> && ls preprints | head -5; d=$(ls -d preprints/*/ | head -1); echo "$d"; ls "$d"; find preprints -maxdepth 2 \( -iname 'README*' -o -iname 'Makefile' -o -iname '*build*' \) | head -20; grep -ril "build" preprints --include=*.md --include=*.txt 2>/dev/null | head; which lake elan lean latexmk pdflatex 2>&1
cd <host-path> && cat README.md; echo ----; ls -la build; echo ----; cat build/* 2>/dev/null | head -60
cd <host-path> && grep -rl --include=README.md -E '^\s*(python3?|pip|make|pdflatex|latexmk|lake|sh )' . | head -30; echo ----; cat "preprints/Complex-Matrix-Multiplication-Below-2.258-and-Rectangular-Bounds-September-24-2026/verification/README.md" | head -80
cd "<host-path>" && ls verification verification/scripts 2>&1; cat README.md | head -60; echo ----; python3 -B verification/scripts/check_certificate.py; echo "rc=$?"
P="<host-path>"; W=<host-path> +%s); mkdir -p "$W" && cp -R "$P/verification/certificates" "$W/certificates" && cd "$P" && time python3 -B "$W/certificates/checks/run_checks.py"; echo "rc=$?"; echo "W=$W"
cd <host-path> && mkdir -p .coldpass && cat > .coldpass/probe.json <<'EOF'
{"kind":"cli","argv":["/usr/bin/python3","-B","<host-path>"}
EOF
cat > .coldpass/report.json <<'EOF'
{"outcome":"working","operation":"Ran the default rectangular certificate checks from preprints/Complex-Matrix-Multiplication-Below-2.258-and-Rectangular-Bounds-September-24-2026/verification/README.md (python3 -B <copy>/certificates/checks/run_checks.py). It runs verify_rational.py, verify_construction.py and verify_baseline.py, compares outputs byte-for-byte, and reports status passed.","operation_scope":"Covers only the default standard-library rectangular checks. Not exercised: the square-certificate replay (reproduce.py and verify_manuscript_bounds.py), which needs a C++17 compiler with int128, UBSan and GMP headers; the optional --numerics diagnostics, which need mpmath==1.3.0; and the Lean/Comparator checks under lean/. The tensor constructions and analytic allowances are not checked by this command.","steps":["Read README.md and lean/README.md; no runnable command at top level. lean/ComparatorChallenges/README.md needs comparator, landrun and lean4export plus lake, and was not attempted.","Checked preprint verification READMEs for a runnable command; chose the Complex-Matrix-Multiplication preprint.","python3 -B verification/scripts/check_certificate.py  -> PASS: 22 numerical payload entries; 57 clean build inputs (rc 0; integrity-only, not used as probe)","cp -R verification/certificates <host-path> (fresh writable copy, as the README requires)","python3 -B <host-path>  -> status passed, rc 0"],"deviations":["Used a fixed copy path under <host-path> instead of the README's mktemp -d, because the probe argv must be a literal path. The copy was made outside the workspace so the repo is not written to.","No dependencies were installed; the default checks need only python3 3.12, which is present."],"stuck_step":null,"main_blocker":null,"suggested_fix":null}
EOF
<env-var>=<host-path> python3 -I <host-path> --validate; echo "rc=$?"
cd "<host-path>" && ls -la verification/certificates/checks/; ls "<host-path>"; grep -n "write\|open(" verification/certificates/checks/run_checks.py | head -20
P=<host-path>; cd "$P" && python3 -B verification/certificates/checks/run_checks.py | tail -3; echo "rc=${PIPESTATUS[0]}"; rm -rf <host-path>; cd <host-path> && cat > .coldpass/probe.json <<EOF
{"kind":"cli","argv":["/usr/bin/python3","-B","$P/verification/certificates/checks/run_checks.py"],"cwd":"$P"}
EOF
sed -i 's|Ran the default rectangular certificate checks from .*|Ran the default rectangular certificate checks from preprints/Complex-Matrix-Multiplication-Below-2.258-and-Rectangular-Bounds-September-24-2026/verification/README.md (python3 -B verification/certificates/checks/run_checks.py). It runs verify_rational.py, verify_construction.py and verify_baseline.py, compares outputs byte-for-byte, and reports status passed. It writes execution_report.json beside the inputs.","operation_scope"|' .coldpass/report.json; sed -i 's|Used a fixed copy path under <host-path> the checks in place instead of in the README'"'"'s fresh copy. The probe argv must reference the repo script directly; a copy outside the repo was rejected as agent-written code. The runner only adds execution_report.json. The temporary copy was removed.",|' .coldpass/report.json; python3 -m json.tool .coldpass/report.json >/dev/null && python3 -m json.tool .coldpass/probe.json && <env-var>=<host-path> python3 -I <host-path> --validate; echo "rc=$?"
P=<host-path>; cd <host-path> && cat > .coldpass/probe.json <<EOF
{"kind":"cli","argv":["/usr/bin/python3","-B","$P/verification/certificates/checks/run_checks.py"],"cwd":"$P"}
EOF
cat > .coldpass/report.json <<'EOF'
{"outcome":"working","operation":"Ran the default rectangular certificate checks from preprints/Complex-Matrix-Multiplication-Below-2.258-and-Rectangular-Bounds-September-24-2026/verification/README.md (python3 -B verification/certificates/checks/run_checks.py). It runs verify_rational.py, verify_construction.py and verify_baseline.py, compares outputs byte-for-byte with the saved JSON, and reports status passed (exit 0).","operation_scope":"Covers only the default standard-library rectangular checks. Not exercised: the square-certificate replay (reproduce.py and verify_manuscript_bounds.py), which needs a C++17 compiler with int128, UBSan and GMP headers; the optional --numerics diagnostics, which need mpmath==1.3.0; and the Lean/Comparator checks under lean/, which need comparator, landrun, lean4export and lake. The command checks finite arithmetic outputs only; the tensor constructions and analytic allowances are not verified.","steps":["Read README.md and lean/README.md; no runnable command at top level. lean/ComparatorChallenges/README.md needs comparator, landrun, lean4export and lake; not attempted.","Checked preprint verification READMEs for a runnable command; chose the Complex-Matrix-Multiplication preprint.","cd preprints/Complex-Matrix-Multiplication-Below-2.258-and-Rectangular-Bounds-September-24-2026 && python3 -B verification/scripts/check_certificate.py  -> PASS: 22 numerical payload entries; 57 clean build inputs (rc 0; integrity-only, not used as probe)","python3 -B verification/certificates/checks/run_checks.py  -> status passed, rc 0"],"deviations":["Ran the checks in place instead of in the README's fresh writable copy. The probe argv must reference the repo script directly, and a copy outside the repo was rejected as agent-written code. The runner only adds execution_report.json beside the inputs; the temporary copy I made earlier was removed.","No dependencies were installed; the default checks need only python3 3.12, which is present on the guest."],"stuck_step":null,"main_blocker":null,"suggested_fix":null}
EOF
python3 -m json.tool .coldpass/report.json >/dev/null && python3 -m json.tool .coldpass/probe.json >/dev/null && echo JSON_OK && <env-var>=<host-path> python3 -I <host-path> --validate; echo "rc=$?"