vinnylarouge/jevlike

No problems traced to this repository code in these checks.

OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.

A short run cannot establish that code is safe.

Connections show IP addresses; proxies can hide the final hostname.

Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.

Full safety evidence →

• Evaluation prints top-1 accuracy on the held-out test file: Real input produced the expected result: "top1": 0.9975

  Source: The evaluation prints top-1 accuracy, which is the fraction of correct first choices.

  Input: runs/synthetic.pt data/synthetic/test.jsonl

  Expected: model top1 printed as 0.9975

  Command: <guest-path> runs/synthetic.pt data/synthetic/test.jsonl

  Observed: "top1": 0.9975, "top3": 1.0, "ece": 0.004109287678147666, "examples": 400 }, "shuffled_context": { "top1": 0.2275, "top3": 0.6825, "ece": 0.6250180058996193, "examples": 400 }, "device": "cpu" }

• Score one new menu with the trained model: Real input produced the expected result: amber badger

  Source: They create local synthetic data, train on it, evaluate the saved model and score one new menu.

  Input: context about amber badger with three options

  Expected: amber badger receives probability near 1.0

  Command: <guest-path> runs/synthetic.pt --context 'Choose the exact badge amber badger. Badge: amber badger.' --option 'azure crane' --option 'amber badger' --option 'gold heron'

  Observed: amber badger", "probability": 1.0 }, { "option": "gold heron", "probability": 4.9812562963325036e-08 } ]

• Shuffled-context control is reported alongside the model: Real input produced the expected result: "top1": 0.2275

  Source: The command also prints a shuffled-context control, which pairs each menu with the wrong context.

  Input: runs/synthetic.pt data/synthetic/test.jsonl

  Expected: shuffled_context top1 printed as 0.2275

  Command: <guest-path> runs/synthetic.pt data/synthetic/test.jsonl

  Observed: "top1": 0.2275, "top3": 0.6825, "ece": 0.6250180058996193, "examples": 400 }, "device": "cpu" }

Examples

• Quickstart training on synthetic data writes a checkpoint: command completed successfully

  Source: README.md Quickstart

  Input: data/synthetic/train.jsonl with validation; fresh output runs/replay-train.pt

  Expected: checkpoint written with best_validation_nll

  Command: <guest-path> data/synthetic/train.jsonl --validation data/synthetic/validation.jsonl --output runs/replay-train.pt

  Observed: h": 4, "train_nll": 0.042619120180606845, "validation_nll": 0.0345140016078949, "device": "cpu"} {"epoch": 5, "train_nll": 0.019992097705602646, "validation_nll": 0.028758792504668236, "device": "cpu"} {"epoch": 6, "train_nll": 0.021846030103042723, "validation_nll": 0.019645809847861528, "device": "cpu"} {"epoch": 7, "train_nll": 0.009171441148966552, "validation_nll": 0.007989753340370952, "device": "cpu"} {"epoch": 8, "train_nll": 0.003976261492818594, "validation_nll": 0.008053538376116194, "device": "cpu"} {"checkpoint": "runs/replay-train.pt", "best_validation_nll": 0.007989753340370952}

• Doom example plays the released joint checkpoint and writes a trace: command completed successfully

  Source: README.md Demo

  Input: examples/checkpoints/joint-imitation.pt, 10 episodes, 640x480 capture

  Expected: JSON summary with episodes 10 and video/trace paths

  Command: <guest-path> examples/doom/play.py examples/checkpoints/joint-imitation.pt --episodes 10 --game-seconds 35.3 --device cpu --capture-resolution 640x480 --output runs/doom.mp4 --trace runs/doom-trace.json

  Observed: {"episodes": 10, "mean_reward": -94.78393249511718, "mean_kills": 0.4, "mean_wall_game_ratio": 1.0215097106955706, "online_inference_ms": 2.748335302327975, "cpu_ms": 1.0682277600000134, "video": "runs/doom.mp4", "trace": "runs/doom-trace.json"}

• Doom example smoke tests: command completed successfully

  Source: AGENTS.md Setup test commands

  Input: examples/doom/test_smoke.py

  Expected: 2 passed

  Command: <guest-path> -q examples/doom/test_smoke.py

  Observed: .. [100%] 2 passed in 5.91s

• Chess example smoke test: command completed successfully

  Source: AGENTS.md Setup test commands

  Input: examples/chess/test_smoke.py

  Expected: ok

  Command: <guest-path> test_smoke.py

  Observed: ok

• Repository offline test suite: command completed successfully

  Source: AGENTS.md Setup test commands

  Input: tests/test_smoke.py

  Expected: 1 passed

  Command: <guest-path> -q

  Observed: . [100%] 1 passed in 11.19s

• Film rendering from the Doom trace: not run: needs Node/npm (absent in guest) and a separate npm/npx setup step that fetches an extra tool; that step was not performed

  Source: README.md Demo

  Expected: documented command completes successfully

  Observed: not run: needs Node/npm (absent in guest) and a separate npm/npx setup step that fetches an extra tool; that step was not performed

• Wikispeedia next-click data example: not run: scripts/get_wikispeedia.sh downloads external SNAP archives; not needed for the core quickstart

  Source: README.md Wikispeedia example

  Expected: documented command completes successfully

  Observed: not run: scripts/get_wikispeedia.sh downloads external SNAP archives; not needed for the core quickstart

• Frozen Hugging Face encoder training: not run: requires transformers install and downloading Qwen/Qwen2.5-0.5B from Hugging Face (large model download)

  Source: README.md Use a frozen pretrained encoder

  Expected: documented command completes successfully

  Observed: not run: requires transformers install and downloading Qwen/Qwen2.5-0.5B from Hugging Face (large model download)

Local or hosted: local

Train, evaluate and score a one-pass option scorer with the jevlike CLIs from the README Quickstart (jevlike-data, jevlike-train, jevlike-eval, jevlike-predict), plus the documented Doom and chess example checks.

cd <guest-path> && pwd && ls -la && id -u && python3 --version && cat /etc/os-release | head -3 && git status 2>&1 | head -2; ls /inputs 2>&1
cd <guest-path> && find . -maxdepth 2 -not -path './.git*' | head -100; echo ---; cat README.md 2>/dev/null | head -200
cd <guest-path> && cat pyproject.toml; echo ---; which uv git node npm; uv --version 2>&1; git --version; ls .coldpass; echo ---; grep -n -i "python\|uv " AGENTS.md | head -20; ls examples/* | head -50; ls tests
cd <guest-path> && git init -q && git remote add origin https://github.com/vinnylarouge/jevlike && git fetch --depth 1 origin 94f5fd1b0b11d52bbdfdf4e0ee6aa96b568f8452 2>&1 | tail -5; git reset --mixed FETCH_HEAD 2>&1 | tail -3; git log --oneline -1; git status --short | head; python3 -m pip --version 2>&1; python3 -m pip install --user uv 2>&1 | tail -3 ; ls ~/.local/bin
sudo -n apt-get update -qq 2>&1 | tail -2; sudo -n DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3-pip python3-venv 2>&1 | tail -3; python3 -m pip --version && python3 -m pip install --user --break-system-packages -q uv 2>&1 | tail -3; ls ~/.local/bin; ~/.local/bin/uv --version
cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && uv venv 2>&1 | tail -2 && source .venv/bin/activate && uv pip install -e '.[dev]' 2>&1 | tail -5; which jevlike-train; python --version
cd <guest-path> && source .venv/bin/activate && jevlike-data synthetic --output data/synthetic 2>&1 | tail -5; ls data/synthetic; jevlike-train data/synthetic/train.jsonl --validation data/synthetic/validation.jsonl --output runs/synthetic.pt 2>&1 | tail -8; ls -la runs
cd <guest-path> && source .venv/bin/activate && jevlike-eval runs/synthetic.pt data/synthetic/test.jsonl 2>&1 | tail -5; echo "--- predict"; jevlike-predict runs/synthetic.pt --context "Choose the exact badge amber badger. Badge: amber badger." --option "azure crane" --option "amber badger" --option "gold heron" 2>&1 | tail -20
cd <guest-path> && source .venv/bin/activate && jevlike-eval runs/synthetic.pt data/synthetic/test.jsonl 2>&1
cd <guest-path> && source .venv/bin/activate && python -m pytest -q 2>&1 | tail -5; echo ---; grep -rn -i "sleep\|cron\|schedule\|while True\|webhook\|--once\|--now\|interval" --include=*.py --include=*.md --include=*.sh jevlike examples/doom/*.py examples/chess/*.py README.md examples/*/README.md scripts 2>/dev/null | head -30
cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && source .venv/bin/activate && timeout 900 uv pip install -e '.[dev,games]' 2>&1 | tail -6; echo "rc=$?"
cd <guest-path> && source .venv/bin/activate && PYTHONPATH=examples/doom timeout 900 pytest -q examples/doom/test_smoke.py 2>&1 | tail -8; echo "rc=${PIPESTATUS[0]}"; cd examples/chess && timeout 900 python test_smoke.py 2>&1 | tail -8; echo "chess rc=${PIPESTATUS[0]}"
cd <guest-path> && source .venv/bin/activate && ls runs/doom.mp4 runs/doom-trace.json 2>&1; time (timeout 1500 python examples/doom/play.py examples/checkpoints/joint-imitation.pt --episodes 10 --game-seconds 35.3 --device cpu --capture-resolution 640x480 --output runs/doom.mp4 --trace runs/doom-trace.json > /tmp/doom-out.txt 2>&1); echo rc=$?; tail -c 1200 /tmp/doom-out.txt; ls -la runs
cd <guest-path> && ls -la .coldpass runs 2>&1; ls <guest-path> 2>&1
cd <guest-path> && rm -f runs/doom.mp4 runs/doom-trace.json && V=<guest-path> && cat > .coldpass/probe.json <<EOF
{"kind":"cli","argv":["$V/jevlike-predict","runs/synthetic.pt","--context","Choose the exact badge amber badger. Badge: amber badger.","--option","azure crane","--option","amber badger","--option","gold heron"],"cwd":"<guest-path>","expected_text":"amber badger"}
EOF
cat > .coldpass/claims.json <<EOF
[
{"kind":"claim","claim":"Evaluation prints top-1 accuracy on the held-out test file","source":"The evaluation prints top-1 accuracy, which is the fraction of correct first choices.","input":"runs/synthetic.pt data/synthetic/test.jsonl","expected":"model top1 printed as 0.9975","probe_kind":"cli","argv":["$V/jevlike-eval","runs/synthetic.pt","data/synthetic/test.jsonl"],"expected_text":"\"top1\": 0.9975","cwd":"<guest-path>"},
{"kind":"claim","claim":"Score one new menu with the trained model","source":"They create local synthetic data, train on it, evaluate the saved model and score one new menu.","input":"context about amber badger with three options","expected":"amber badger receives probability near 1.0","probe_kind":"cli","argv":["$V/jevlike-predict","runs/synthetic.pt","--context","Choose the exact badge amber badger. Badge: amber badger.","--option","azure crane","--option","amber badger","--option","gold heron"],"expected_text":"amber badger","cwd":"<guest-path>"},
{"kind":"claim","claim":"Shuffled-context control is reported alongside the model","source":"The command also prints a shuffled-context control, which pairs each menu with the wrong context.","input":"runs/synthetic.pt data/synthetic/test.jsonl","expected":"shuffled_context top1 printed as 0.2275","probe_kind":"cli","argv":["$V/jevlike-eval","runs/synthetic.pt","data/synthetic/test.jsonl"],"expected_text":"\"top1\": 0.2275","cwd":"<guest-path>"}
]
EOF
cat > .coldpass/examples.json <<EOF
[
{"kind":"example","claim":"Quickstart training on synthetic data writes a checkpoint","source":"README.md Quickstart","input":"data/synthetic/train.jsonl with validation; fresh output runs/replay-train.pt","expected":"checkpoint written with best_validation_nll","argv":["$V/jevlike-train","data/synthetic/train.jsonl","--validation","data/synthetic/validation.jsonl","--output","runs/replay-train.pt"],"cwd":"<guest-path>","expected_text":"best_validation_nll"},
{"kind":"example","claim":"Doom example plays the released joint checkpoint and writes a trace","source":"README.md Demo","input":"examples/checkpoints/joint-imitation.pt, 10 episodes, 640x480 capture","expected":"JSON summary with episodes 10 and video/trace paths","argv":["$V/python","examples/doom/play.py","examples/checkpoints/joint-imitation.pt","--episodes","10","--game-seconds","35.3","--device","cpu","--capture-resolution","640x480","--output","runs/doom.mp4","--trace","runs/doom-trace.json"],"cwd":"<guest-path>","expected_text":"\"episodes\": 10"},
{"kind":"example","claim":"Doom example smoke tests","source":"AGENTS.md Setup test commands","input":"examples/doom/test_smoke.py","expected":"2 passed","argv":["$V/pytest","-q","examples/doom/test_smoke.py"],"env":{"PYTHONPATH":"examples/doom"},"cwd":"<guest-path>","expected_text":"2 passed"},
{"kind":"example","claim":"Chess example smoke test","source":"AGENTS.md Setup test commands","input":"examples/chess/test_smoke.py","expected":"ok","argv":["$V/python","test_smoke.py"],"cwd":"<guest-path>","expected_text":"ok"},
{"kind":"example","claim":"Repository offline test suite","source":"AGENTS.md Setup test commands","input":"tests/test_smoke.py","expected":"1 passed","argv":["$V/pytest","-q"],"cwd":"<guest-path>","expected_text":"1 passed"},
{"kind":"example","claim":"Film rendering from the Doom trace","source":"README.md Demo","reason":"not run: needs Node/npm (absent in guest) and a separate npm/npx setup step that fetches an extra tool; that step was not performed","deferred":true},
{"kind":"example","claim":"Wikispeedia next-click data example","source":"README.md Wikispeedia example","reason":"not run: scripts/get_wikispeedia.sh downloads external SNAP archives; not needed for the core quickstart","deferred":true},
{"kind":"example","claim":"Frozen Hugging Face encoder training","source":"README.md Use a frozen pretrained encoder","reason":"not run: requires transformers install and downloading Qwen/Qwen2.5-0.5B from Hugging Face (large model download)","deferred":true}
]
EOF
cat > .coldpass/cycles.json <<EOF
{"detected":false,"reason":"Entry points inspected (jevlike-data, jevlike-train, jevlike-eval, jevlike-predict in pyproject.toml; jevlike/*.py; examples). No cron, scheduler, --once/--now, loop or webhook exists in the product. The only sleep calls are game-pacing loops in examples/doom/play.py and examples/chess/play.py, which are not product cycles. Each CLI runs once when invoked.","checks":[]}
EOF
for f in probe claims examples cycles; do python3 -m json.tool .coldpass/$f.json >/dev/null && echo "$f ok"; done; ls runs
cd <guest-path> && <env-var>=<guest-path> timeout 1500 python3 -I <guest-path> --validate --replay 2>&1 | tail -80; echo "rc=${PIPESTATUS[0]}"
cd <guest-path> && cat > .coldpass/report.json <<'EOF'
{"outcome":"working","fault_owner":null,"operation":"Train, evaluate and score a one-pass option scorer with the jevlike CLIs from the README Quickstart (jevlike-data, jevlike-train, jevlike-eval, jevlike-predict), plus the documented Doom and chess example checks.","operation_scope":"Linux Ubuntu 24.04, Python 3.12.3, CPU only, pinned commit 94f5fd1b0b11d52bbdfdf4e0ee6aa96b568f8452. Installed uv 0.13.5 via pip (user install) and python3-pip/python3-venv via apt. Game extras installed via uv. No Node/npm, no external datasets or Hugging Face models.","steps":["git init; git remote add origin https://github.com/vinnylarouge/jevlike; git fetch --depth 1 origin 94f5fd1b0b11d52bbdfdf4e0ee6aa96b568f8452; git reset --mixed FETCH_HEAD","sudo apt-get install -y python3-pip python3-venv","python3 -m pip install --user --break-system-packages uv","uv venv","source .venv/bin/activate && uv pip install -e '.[dev]'","jevlike-data synthetic --output data/synthetic","jevlike-train data/synthetic/train.jsonl --validation data/synthetic/validation.jsonl --output runs/synthetic.pt","jevlike-eval runs/synthetic.pt data/synthetic/test.jsonl","jevlike-predict runs/synthetic.pt --context \"Choose the exact badge amber badger. Badge: amber badger.\" --option \"azure crane\" --option \"amber badger\" --option \"gold heron\"","uv pip install -e '.[dev,games]' (AGENTS.md setup)","pytest -q; PYTHONPATH=examples/doom pytest -q examples/doom/test_smoke.py; cd examples/chess && python test_smoke.py","python examples/doom/play.py examples/checkpoints/joint-imitation.pt --episodes 10 --game-seconds 35.3 --device cpu --capture-resolution 640x480 --output runs/doom.mp4 --trace runs/doom-trace.json"],"deviations":["uv was not present; installed via pip (apt python3-pip first, since pip was missing). Setup only.","Quickstart eval: top1 0.9975 for the model vs 0.2275 for the shuffled-context control. Predict gave 'amber badger' probability 1.0.","Doom play run: 10 episodes gave mean_kills 0.4 and mean_reward -94.78; the README cites 0.60 kills and -97.50 reward for the same checkpoint. Results vary between runs, so this is not treated as a mismatch.","README Demo film step not run: it needs Node/npm, which is absent, and a separate tool download that this guest does not permit. This is a prerequisite gap for that step only; the core operation is unaffected.","Wikispeedia and frozen Hugging Face encoder examples deferred: they download external data or a large model; not needed for the core operation.","The README's frozen-encoder path was not exercised, so its Hugging Face download behaviour is unverified."],"stuck_step":null,"main_blocker":null,"plain_failure":null,"suggested_fix":"None needed for the core operation. To run the README film step, provide Node/npm and allow its separate tool download."}
EOF
python3 -m json.tool .coldpass/report.json >/dev/null && echo report ok; ls -la .coldpass
cd <guest-path> && <env-var>=<guest-path> timeout 1500 python3 -I <guest-path> --validate --replay > /tmp/final-check.txt 2>&1; echo "rc=$?"; python3 -I -c "import json;d=json.load(open('/tmp/final-check.txt'));print(d['valid'],d['problems'],d['replay']['status'])" 2>&1 | tail -3