autogluon/tabarena

Review signals: 1 hardcoded secret.

Gitleaks found a possible hardcoded secret; value withheld. packages/tabflow_slurm/BENCHMARK_LOG.md

OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.

A short run cannot establish that code is safe.

Connections show IP addresses; proxies can hide the final hostname.

Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.

Full safety evidence →

• TabArena computes a leaderboard from results.: Real input produced the expected result: === Leaderboard ===

  Source: TabArena ranks **models** (one method, tuned by TabArena under a shared protocol) and **systems**

  Input: synthetic example results generated by the documented script

  Expected: prints a leaderboard ranking methods

  Command: .venv/bin/python examples/plots/run_generate_custom_leaderboard.py

  Observed: === Leaderboard === elo elo+ elo- rank winrate improvability improvability+ improvability- mrr metric_error time_train_s time_infer_s median_rank median_improvability median_loss_rescaled median_metric_error median_time_train_s median_time_infer_s method ModelA 1161.2 60.7 47.9 1.80 0.733333 0.02753

Examples

• Generate a custom leaderboard from synthetic results.: command completed successfully

  Source: examples/README.md, Generating Plots and Leaderboards

  Input: documented custom leaderboard script

  Expected: prints leaderboard, Elo and ranks

  Command: .venv/bin/python examples/plots/run_generate_custom_leaderboard.py

  Observed: 0.750011 0.296726 24.270589 1.759914 ModelB 1000.0 35.8 57.7 2.65 0.450000 0.502097 17.851416 1.521589 2.666667 0.064467 0.458040 0.283038 16.516307 1.539750 === Pairwise win-rate matrix (results_per_task) === ModelA ModelC ModelB ModelD ModelA NaN 70.000000 78.300003 71.699997 ModelC 30.000000 NaN 46.700001 63.299999 ModelB 21.700001 53.299999 NaN 60.000000 ModelD 28.299999 36.700001 40.000000 NaN

Local or hosted: both

Generate a custom benchmark leaderboard

pwd && rg --files -g 'README*' -g 'pyproject.toml' -g 'package.json' -g 'Makefile' -g '.github/workflows/**' | head -80 && git status --short && python3 --version && sed -n '1,240p' README.md
ls -la; find . -maxdepth 3 -type f | head -80; python3 --version; sed -n '1,240p' README.md
command -v uv || true; command -v pip || true; sed -n '1,180p' examples/benchmarking/run_quickstart_tabarena_model.py; cat pyproject.toml; sed -n '1,180p' examples/README.md
python3 -m venv .venv && .venv/bin/python -m pip --version && uv venv --seed --python 3.12 .venv
uv pip install --python .venv/bin/python --prerelease=allow -e './packages/tabarena[plot]'
mkdir -p .coldpass && .venv/bin/python examples/plots/run_generate_custom_leaderboard.py
mkdir -p .coldpass && cat > .coldpass/probe.json <<'EOF'
{"kind":"cli","argv":[".venv/bin/python","examples/plots/run_generate_custom_leaderboard.py"],"cwd":"<guest-path>","expected_text":"=== Leaderboard with a metric of your own ==="}
EOF
cat > .coldpass/claims.json <<'EOF'
[{"kind":"claim","claim":"The custom leaderboard example computes ranks and Elo from a results table.","source":"Compute a leaderboard (ranks, Elo, win-rates) from your own results table with `bencheval` - a no-download starting point for the evaluation API.","input":"synthetic example results generated by the documented script","expected":"leaderboard with Elo and rank columns","probe_kind":"cli","argv":[".venv/bin/python","examples/plots/run_generate_custom_leaderboard.py"],"expected_text":"=== Leaderboard ===","cwd":"<guest-path>"}]
EOF
cat > .coldpass/examples.json <<'EOF'
[{"kind":"example","claim":"Generate a custom leaderboard from synthetic results.","source":"examples/README.md, Generating Plots and Leaderboards","input":"documented custom leaderboard script","expected":"prints leaderboard, Elo and ranks","argv":[".venv/bin/python","examples/plots/run_generate_custom_leaderboard.py"],"cwd":"<guest-path>","expected_text":"=== Leaderboard ==="}]
EOF
cat > .coldpass/cycles.json <<'EOF'
{"detected":false,"reason":"Inspected README quickstart, examples index, and repository scripts; product behavior is invoked as finite benchmark or leaderboard commands, with no product scheduler or recurring cycle." ,"checks":[]}
EOF
find <guest-path> -maxdepth 2 -name 'check.py' -o -name 'coldpass-preflight.py'
<env-var>=<guest-path> python3 -I <guest-path> --validate --replay
cat > .coldpass/claims.json <<'EOF'
[{"kind":"claim","claim":"The custom leaderboard example generates a leaderboard.","source":"Compute a leaderboard","input":"synthetic example results generated by the documented script","expected":"leaderboard with Elo and rank columns","probe_kind":"cli","argv":[".venv/bin/python","examples/plots/run_generate_custom_leaderboard.py"],"expected_text":"=== Leaderboard ===","cwd":"<guest-path>"}]
EOF

<env-var>=<guest-path> python3 -I <guest-path> --validate --replay
cat > .coldpass/claims.json <<'EOF'
[{"kind":"claim","claim":"TabArena computes a leaderboard from results.","source":"TabArena ranks **models** (one method, tuned by TabArena under a shared protocol) and **systems**","input":"synthetic example results generated by the documented script","expected":"prints a leaderboard ranking methods","probe_kind":"cli","argv":[".venv/bin/python","examples/plots/run_generate_custom_leaderboard.py"],"expected_text":"=== Leaderboard ===","cwd":"<guest-path>"}]
EOF
<env-var>=<guest-path> python3 -I <guest-path> --validate --replay
cat > .coldpass/report.json <<'EOF'
{"outcome":"working","fault_owner":null,"operation":"Generate a custom benchmark leaderboard","operation_scope":"README compatible Python 3.12.3 environment with uv and the plot extra installed","steps":["Created Python 3.12 virtual environment with pip and uv","Installed documented evaluation-only dependency path: uv pip install --prerelease=allow -e './packages/tabarena[plot]'","Ran .venv/bin/python examples/plots/run_generate_custom_leaderboard.py","Validated and replayed probe, README claim, example and cycle evidence successfully"],"deviations":["The quickstart benchmark model requires costly model dependencies and downloaded dataset assets; used the documented no-download custom leaderboard example instead."],"stuck_step":null,"main_blocker":null,"plain_failure":null,"suggested_fix":null}
EOF