cosmin-novac/memry

Review signals: 3 download and execute, 34 known vulnerability.

Reviewed as expected for a test run: attempted writes that failed, so nothing was written.

Source contains download and execute; review its purpose. deploy/install.sh 3 findings.

OSV reports a known vulnerability in this dependency version. 3 distinct locations. 34 findings.

OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.

A short run cannot establish that code is safe.

Connections show IP addresses; proxies can hide the final hostname.

Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.

Full safety evidence →

• The built-in evaluation harness scores retrieval deterministically and offline.: Real input produced the expected result: recall@5: 1.000

  Source: The harness ingests each case through the full write path, then scores retrieval

  Input: evals/datasets/synthetic_v1.jsonl -k 5

  Expected: recall@5: 1.000

  Command: /tmp/memry-venv/bin/memry eval --dataset evals/datasets/synthetic_v1.jsonl -k 5

  Observed: recall@5: 1.000 MRR: 0.814 search p50: 0.7 ms p95: 1.0 ms llm: none embedder: hash:v1-256

Examples

• Run the synthetic retrieval evaluation: command completed successfully

  Source: README.md / Evaluation

  Input: synthetic_v1.jsonl, k=5

  Expected: retrieval metrics

  Command: /tmp/memry-venv/bin/memry eval --dataset evals/datasets/synthetic_v1.jsonl -k 5

  Observed: dataset: evals/datasets/synthetic_v1.jsonl cases: 8 questions: 17 memories: 19 recall@5: 1.000 MRR: 0.814 search p50: 0.7 ms p95: 0.8 ms llm: none embedder: hash:v1-256

Local or hosted: local

Run the synthetic retrieval evaluation

pwd; rg --files -g 'README*' -g 'pyproject.toml' -g 'package.json' -g 'Makefile' -g 'AGENTS.md' -g '*.toml' -g '*.yml' -g '*.yaml' -g 'requirements*.txt' -g '*.json' | head -80
ls -la; find . -maxdepth 3 -type f | sort | head -100; python3 --version
cat README.md; cat pyproject.toml
python3 -m venv /tmp/memry-venv && /tmp/memry-venv/bin/python -m pip install -e '.[dev]'
sed -n '1,220p' src/memry/cli.py; sed -n '1,100p' evals/datasets/synthetic_v1.jsonl; mkdir -p .coldpass
sed -n '1,200p' src/memry/config.py; grep -n 'eval\|dataset' README.md
MEMRY_DB_PATH=/tmp/memry-run.db /tmp/memry-venv/bin/memry eval --dataset evals/datasets/synthetic_v1.jsonl -k 5
cat > .coldpass/probe.json <<'EOF'
{"kind":"cli","argv":["/tmp/memry-venv/bin/memry","eval","--dataset","evals/datasets/synthetic_v1.jsonl","-k","5"],"cwd":"<guest-path>","env":{"MEMRY_DB_PATH":"/tmp/memry-probe.db"},"expected_text":"recall@5:     1.000"}
EOF
cat > .coldpass/claims.json <<'EOF'
[{"kind":"claim","claim":"The built-in evaluation harness scores retrieval deterministically and offline.","source":"The harness ingests each case through the full write path, then scores retrieval","input":"evals/datasets/synthetic_v1.jsonl -k 5","expected":"recall@5:     1.000","probe_kind":"cli","argv":["/tmp/memry-venv/bin/memry","eval","--dataset","evals/datasets/synthetic_v1.jsonl","-k","5"],"expected_text":"recall@5:     1.000","cwd":"<guest-path>","env":{"MEMRY_DB_PATH":"/tmp/memry-claim.db"}}]
EOF
cat > .coldpass/examples.json <<'EOF'
[{"kind":"example","claim":"Run the synthetic retrieval evaluation","source":"README.md / Evaluation","input":"synthetic_v1.jsonl, k=5","expected":"retrieval metrics","argv":["/tmp/memry-venv/bin/memry","eval","--dataset","evals/datasets/synthetic_v1.jsonl","-k","5"],"cwd":"<guest-path>","env":{"MEMRY_DB_PATH":"/tmp/memry-example.db"},"expected_text":"MRR:"}]
EOF
cat > .coldpass/cycles.json <<'EOF'
{"detected":false,"checks":[],"reason":"Inspected README quickstart and CLI entrypoints; they document explicit add/search/eval operations and no cron, scheduled loop, webhook, --once, or --now cycle."}
EOF
<env-var>=<guest-path> python3 -I <guest-path> --validate --replay
cat > .coldpass/report.json <<'EOF'
{"outcome":"working","fault_owner":null,"operation":"Run the synthetic retrieval evaluation","operation_scope":"Local offline evaluation using the built-in dataset, SQLite, and hash embeddings; Python 3.12.3 in a venv","steps":["Read README.md and pyproject.toml; confirmed Python >=3.11.","Created /tmp/memry-venv and installed the documented development package with pip install -e '.[dev]'.","Ran memry eval --dataset evals/datasets/synthetic_v1.jsonl -k 5; observed 8 cases, 17 questions, recall@5 1.000, and MRR 0.814.","Replayed probe, README claim, and documented evaluation example with the coldpass checker; validation passed."],"deviations":[],"stuck_step":null,"main_blocker":null,"plain_failure":null,"suggested_fix":null}
EOF
cat .coldpass/report.json; ls -la .coldpass