{ "checker": { "claims": [ { "claim": "install writes the guard hooks into the agent's user config", "command": "jev-guard install claude", "evidence": "jev-guard: written to \nNo API key found yet: run `jev-guard key ` (or export JEV_API_KEY / AI_GATEWAY_API_KEY). Until then the guard fails open.\n", "expected": "settings.json written under the scratch HOME", "input": "jev-guard install claude with HOME set to a scratch directory", "kind": "claim", "outcome": "Real input produced the expected result: jev-guard: written to", "returncode": 0, "source": "`install` is idempotent and writes the absolute path of the current `node`", "source_checked": true, "source_file": "README.md", "verdict": "yes" }, { "claim": "check exits with a verdict code", "command": "", "evidence": "needs a Jev API key (TypeSafe, Vercel AI Gateway or OpenRouter); none is available in this guest, so the CLI exits 3 with 'no credentials'", "expected": "", "input": "", "kind": "claim", "outcome": "needs a Jev API key (TypeSafe, Vercel AI Gateway or OpenRouter); none is available in this guest, so the CLI exits 3 with 'no credentials'", "source": "Assess one tool call; exit 0 allow, 1 ask, 2 deny", "source_checked": true, "source_file": "README.md", "verdict": "blocked" } ], "cycle_checks": [ { "claim": "Scheduled or looping main behavior", "command": "", "evidence": "Entry points inspected: src/cli.js commands (hook, acp, check, scan, scan-skills, install, key), README CLI section, hooks/*.json. jev-guard runs as a per-tool-call hook invoked by the host agent, with no cron, scheduler, loop, --once/--now or webhook; src/session.js notes 'no scheduler'. No product cycle to trigger.", "expected": "identify the product trigger", "input": "source inspection", "kind": "cycle", "outcome": "Entry points inspected: src/cli.js commands (hook, acp, check, scan, scan-skills, install, key), README CLI section, hooks/*.json. jev-guard runs as a per-tool-call hook invoked by the host agent, with no cron, scheduler, loop, --once/--now or webhook; src/session.js notes 'no scheduler'. No product cycle to trigger.", "source": "repository entry points", "verdict": "yes" } ], "details": { "checks": [ { "check": "install", "details": { "argv": [ "jev-guard", "install", "claude" ], "kind": "cli", "needs_browser": false, "needs_credentials": true, "returncode": 0, "scope": "partial: install", "stderr": "", "stdout": "jev-guard: written to \nNo API key found yet: run `jev-guard key ` (or export JEV_API_KEY / AI_GATEWAY_API_KEY). Until then the guard fails open.", "working": false }, "optional": false, "status": "PASS" }, { "check": "credentials_error", "details": { "argv": [ "jev-guard", "check", "Bash", "{\"command\":\"rm -rf ~/\"}" ], "kind": "cli", "needs_browser": false, "needs_credentials": true, "returncode": 3, "scope": "partial: missing credentials", "stderr": "jev-guard: no credentials: run `jev-guard key ` or set JEV_API_KEY / AI_GATEWAY_API_KEY (exit 3)", "stdout": "", "working": false }, "optional": false, "status": "PASS" } ], "scope": "partial checks", "working": false }, "evidence_strength": "partly_checked", "example_checks": [ { "claim": "offline test suite (README Development section)", "command": "npm test", "deferred": false, "evidence": " duration_ms: 3666.697281\n type: 'test'\n ...\n# Subtest: issues \\#1\u2013\\#3: .claude/docs routing, sub-agent results scanned, thresholds range-checked\nok 17 - issues \\#1\u2013\\#3: .claude/docs routing, sub-agent results scanned, thresholds range-checked\n ---\n duration_ms: 0.423151\n type: 'test'\n ...\n# Subtest: opencode: skill results are checked as instruction files (\\#2)\nok 18 - opencode: skill results are checked as instruction files (\\#2)\n ---\n duration_ms: 5.454866\n type: 'test'\n ...\n1..18\n# tests 18\n# suites 0\n# pass 18\n# fail 0\n# cancelled 0\n# skipped 0\n# todo 0\n# duration_ms 8028.744001", "expected": "18 of 18 tests pass", "input": "npm test in a copy of the checkout named jev-guard (the snapshot path 'workspace' makes one install-path test fail)", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "README.md Development: npm test", "verdict": "yes" }, { "claim": "classify a tool call (README Install quickstart)", "command": "", "deferred": false, "evidence": "needs a Jev API key; none provided. The documented call was run and exits 3 with 'no credentials' (see probe.json credentials_error)", "expected": "documented command completes successfully", "input": "", "kind": "example", "outcome": "needs a Jev API key; none provided. The documented call was run and exits 3 with 'no credentials' (see probe.json credentials_error)", "source": "README.md Install: jev-guard check Bash '{\"command\":\"rm -rf ~/\"}'", "verdict": "blocked" } ], "status": "PASS" }, "deviations": [ "Ubuntu 24.04 apt nodejs is below the required Node 20.3; installed Node 22 from NodeSource (setup_22.x) instead.", "npm i -g jev-guard as non-root failed with EACCES on /usr/lib/node_modules; reran the same README command with sudo -E.", "npm test in 17/18 passed. The failing test (install writes valid config for every target) asserts the hook config contains the string 'jev-guard'; the hook command embeds the checkout path, which here ends in 'workspace', not 'jev-guard'. A copy at /tmp/jevcheck/jev-guard passes 18/18. This is a path artifact of the snapshot directory, not a product defect.", "No Jev credential provided or obtained; the documented check/scan operations were not exercised live (credentials_error recorded instead).", "The checker's replay of the install claim wrote its settings.json under (the checker's HOME override was not applied); this is outside the workspace." ], "fault_owner": "prerequisite", "main_blocker": "No Jev API key (JEV_API_KEY, AI_GATEWAY_API_KEY, or jev-guard key) is available in this guest, and the product cannot judge a tool call without one.", "operation": "Run jev-guard's documented safety check on a tool call (jev-guard check Bash '{\"command\":\"rm -rf ~/\"}') and get a allow/ask/deny verdict from the Jev model", "operation_scope": "Installed the published npm package globally on Ubuntu 24.04 (Node 22.23.4, npm 10.9.9). Ran the local, credential-free parts: install claude into a scratch HOME (works), and the offline test suite (18/18 pass in a copy named jev-guard). The live Jev verdict could not run because no Jev API key (TypeSafe, Vercel AI Gateway or OpenRouter) is available in this guest.", "outcome": "needs_credentials", "plain_failure": "The guard's main job, asking the Jev model whether a command is safe, could not be run live because the product needs a Jev account key and none was provided. Install, the local hook configuration, and the offline test suite work.", "steps": [ "git init; git remote add origin https://github.com/leepokai/jev-guard; git fetch --depth 1 origin cda06c49ceaac207904142ec92bfab793bbd18b1; git reset --mixed FETCH_HEAD", "README Install prerequisite: node >= 20.3 (package.json engines). Ubuntu default nodejs is too old; installed Node 22 via NodeSource setup_22.x and apt-get install nodejs (v22.23.4, npm 10.9.9)", "README Install: npm i -g jev-guard (run as sudo -E npm i -g jev-guard, see deviations); jev-guard installed at /usr/bin/jev-guard", "README CLI: jev-guard check Bash '{\"command\":\"rm -rf ~/\"}' -> exit 3, 'jev-guard: no credentials: run `jev-guard key ` or set JEV_API_KEY / AI_GATEWAY_API_KEY'", "README Install: jev-guard install claude (HOME set to a scratch dir) -> 'jev-guard: written to .../.claude/settings.json'", "README Development: npm test -> 18 tests, 18 pass when the checkout directory is named jev-guard" ], "stuck_step": "jev-guard check Bash '{\"command\":\"rm -rf ~/\"}' (the product's core operation) exits 3: no Jev API key configured.", "suggested_fix": "Provide a TypeSafe, Vercel AI Gateway or OpenRouter key (export JEV_API_KEY or run jev-guard key) and rerun the check and scan commands from the README. Separately, make the 'install writes valid config' test not depend on the checkout directory name." }