gleanwork/glean-indexing-sdk

Review signals: 31 known vulnerability.

Reviewed as expected for a test run: attempted writes that failed, so nothing was written.

OSV reports a known vulnerability in this dependency version. 8 distinct locations. 31 findings.

OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.

A short run cannot establish that code is safe.

Connections show IP addresses; proxies can hide the final hostname.

Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.

Full safety evidence →

• schema document prints the JSON Schema your transform must produce: Real input produced the expected result: "model": "DocumentDefinition"

  Source: `glean-idx schema document` prints the JSON Schema your `transform()` has to

  Input: schema document

  Expected: DocumentDefinition JSON schema

  Command: <guest-path> schema document

  Observed: …"model": "DocumentDefinition", "schema": { "$defs": { "CommentDefinition": { "description": "Describes a comment on a document", "properties": { "id": { "title": "Id", "type": "string" }, "author": { "anyOf": [ { "$ref": "#/$defs/UserReferenceDefinition" }, { "type": "null" } ], "default": null }, "content": { "anyOf

• mock test runs the connector without touching the network: Real input produced the expected result: "documents": 1

  Source: Test it without touching the network:

  Input: README snippet connector with static data, mock phase

  Expected: 1 document posted to mocked Glean

  Command: <guest-path> test --phase mock --connector complete:create_connector

  Observed: …"documents": 1 } } ] } }

• every command takes --output json for a stable envelope: Real input produced the expected result: "ok": true

  Source: Every command takes `--output json` for a stable envelope, `--yes` to skip

  Input: schema document --output json

  Expected: JSON envelope with ok true

  Command: <guest-path> schema document --output json

  Observed: …"ok": true, "data": { "name": "document", "model": "DocumentDefinition", "schema": { "$defs": { "CommentDefinition": { "description": "Describes a comment on a document", "properties": { "id": { "title": "Id", "type": "string" }, "author": { "anyOf": [ { "$ref": "#/$defs/UserReferenceDefinition" }, { "type": "null" } ], "default": null

Examples

• deploy init generates Docker and Terraform files for a connector project: command completed successfully

  Source: README.md

  Input: deploy init --cloud gcp --connector-factory create_connector into a fresh directory

  Expected: generated_files list including glean_deployment.yaml

  Command: <guest-path> deploy init --cloud gcp --connector-factory create_connector --output-dir <guest-path> -y

  Observed: les.tf", "glean_deployment.yaml", ".env.example", ".dockerignore" ], "next_steps": [ "Edit glean_deployment.yaml \u2014 set cluster_name, region, and registry.", "GCP GKE docs: https://cloud.google.com/kubernetes-engine/docs", "GCP Artifact Registry: https://cloud.google.com/artifact-registry/docs", "cp .env.example .env # fill in connector credentials", "glean-idx datasource configure", "glean-idx deploy build --push", "glean-idx deploy secrets upload", "glean-idx deploy apply", "glean-idx deploy run" ] } }

• repository unit tests (documented mise run test task, expanded to uv run pytest): command completed successfully

  Source: mise.toml [tasks.test]

  Input: tests/

  Expected: all unit tests pass; 4 optional cloud-plugin tests skipped

  Command: uv run pytest -q -p no:logging tests/

  Observed: gs.py:7: could not import 'watchtower': No module named 'watchtower' SKIPPED [1] tests/unit_tests/observability/plugins/aws/test_cloudwatch_metrics.py:5: could not import 'boto3': No module named 'boto3' SKIPPED [1] tests/unit_tests/observability/plugins/gcp/test_cloud_logging.py:7: could not import 'google.cloud.logging_v2': No module named 'google' SKIPPED [1] tests/unit_tests/observability/plugins/gcp/test_cloud_monitoring.py:5: could not import 'google.cloud.monitoring_v3': No module named 'google' 1017 passed, 4 skipped in 36.87s Cannot read termcap database; using dumb terminal settings.

Cycles

• the connector indexing job runs on a recurring schedule: Triggered one cycle by hand (run-once substitute: glean-idx test --phase mock runs the same connector full-mode fetch/transform/upload path once against mocked Glean; normal trigger: Kubernetes CronJob daily at 02:00 UTC (not triggered here; needs GKE/GCP secrets)): { "ok": true, "data": { "connector": "create_connector", "mode": "full", "clients": [ "data_client" ], "phases": [ { "phase": "mock", "status": "ran", "fidelity": "mocked Glean, the connector's own data clients", "posted": { "documents": 1 } } ] } }

  Source: src/glean/indexing/deployment/config.py

  Input: snippet connector project, WIKI_API_TOKEN placeholder

  Expected: one full-mode cycle posts 1 document to mocked Glean

  Command: <guest-path> test --phase mock --connector complete:create_connector

  Observed: { "ok": true, "data": { "connector": "create_connector", "mode": "full", "clients": [ "data_client" ], "phases": [ { "phase": "mock", "status": "ran", "fidelity": "mocked Glean, the connector's own data clients", "posted": { "documents": 1 } } ] } }

Local or hosted: local

Build and run the glean-indexing-sdk CLI from source, generate a connector project from the README quickstart snippet, and run its offline mock test (glean-idx test --phase mock), which fetches, transforms and posts one document to mocked Glean; also glean-idx schema document.

cd <guest-path> && pwd && ls -la && id -u && python3 --version && cat README.md | head -c 6000
echo "$CDP_URL"; curl -s -i --max-time 10 "$CDP_URL/json/version"; echo; echo EXIT=$?
cd <guest-path> && tail -c +6000 README.md | sed 's/data:image[^)]*//g' | head -c 7500; echo; echo ---; cat .python-version; grep -n requires-python pyproject.toml
cd <guest-path> && git --version && git init -q && git remote add origin https://github.com/gleanwork/glean-indexing-sdk && git fetch --depth 1 origin 8b6a809814922ca77bf21f80d9b5bbbc6e1f457f 2>&1 | tail -3; git reset --mixed FETCH_HEAD 2>&1 | tail -2; git log --oneline -1 2>&1; git status --short | head -5
which uv || (curl -LsSf https://astral.sh/uv/install.sh | sh 2>&1 | tail -3); ls ~/.local/bin 2>/dev/null; export PATH=$HOME/.local/bin:$PATH; uv --version
cd <guest-path> && grep -n -A6 '^\[tasks."setup"\]\|^\[tasks.setup\]\|^\[tasks.test\]\|^\[tasks."test"\]' mise.toml | head -40; grep -n 'uv sync\|uv run' mise.toml | head
cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && timeout 900 uv sync --locked --extra dev --extra test --extra lint --extra typing 2>&1 | tail -15; echo EXIT=${PIPESTATUS[0]}
cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && uv run glean-idx --help 2>&1 | head -40; echo ---; uv run glean-idx --version 2>&1 | head -3
cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && timeout 1200 uv run pytest -q -p no:logging tests/ 2>&1 | tail -8; echo EXIT=${PIPESTATUS[0]}
cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && uv run glean-idx validate --help 2>&1 | head -30; echo ---; uv run glean-idx schema --help 2>&1 | head -20; echo ---; uv run glean-idx test --help 2>&1 | head -30
cd <guest-path> && ls snippets snippets/non_streaming | head -40; echo ---; export PATH=$HOME/.local/bin:$PATH; uv run glean-idx schema document 2>&1 | head -c 1200; echo; echo EXIT=${PIPESTATUS[0]}
cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && export WIKI_API_TOKEN=placeholder-for-mock && timeout 600 uv run --project <guest-path> glean-idx test --phase mock --connector complete:create_connector 2>&1 | tail -c 3000; echo; echo EXIT=${PIPESTATUS[0]}
cd <guest-path> && ls docs; export PATH=$HOME/.local/bin:$PATH; uv run glean-idx deploy init --help 2>&1 | head -30; echo ---; uv run glean-idx validate --help >/dev/null 2>&1; grep -rn "glean_deployment.yaml" docs README.md CONTRIBUTING.md 2>/dev/null | head -5
mkdir -p <guest-path> && cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && <guest-path> deploy init --cloud gcp --connector-name companywiki --connector-module complete --connector-factory create_connector --output-dir <guest-path> -y 2>&1 | head -c 2500; echo; echo EXIT=${PIPESTATUS[0]}; ls -la <guest-path> 2>&1
cp <guest-path> <guest-path> && cd <guest-path> && export PATH=$HOME/.local/bin:$PATH && export WIKI_API_TOKEN=placeholder-mock-token && timeout 900 <guest-path> test --phase mock --connector complete:create_connector 2>&1 | head -c 4000; echo; echo EXIT=${PIPESTATUS[0]}
cd <guest-path> && env -u GLEAN_SERVER_URL -u GLEAN_INDEXING_API_TOKEN timeout 120 <guest-path> doctor --output json 2>&1 | head -c 1500; echo; echo EXIT=${PIPESTATUS[0]}; echo ---RUNPY; head -c 1500 run.py; echo; echo ---GREP; cd <guest-path> && grep -rniE "cronjob|schedule|--once|while True|interval" src/glean/indexing --include=*.py | grep -v "^.*#" | head -15
cd <guest-path> && sed -n 30,200p run.py | head -c 5000; echo; echo ---; sed -n 68,78p <guest-path>
cd <guest-path> && V=<guest-path>; env -u GLEAN_SERVER_URL -u GLEAN_INDEXING_API_TOKEN $V doctor --output json ><guest-path> 2>&1; echo DOCTOR_EXIT=$?; $V schema document --output json > <guest-path> 2>&1; echo SCHEMA_EXIT=$?; grep -c '"ok": true' <guest-path>; cd <guest-path> && .venv/bin/glean-idx schema document --output json | head -4; ls -la <guest-path>
cd <guest-path> && V=<guest-path>; P=<guest-path>; cat > probe.json <<EOF
{"kind":"cli","argv":["$V","test","--phase","mock","--connector","complete:create_connector"],"cwd":"$P","env":{"WIKI_API_TOKEN":"placeholder-mock-token"},"expected_text":"\"documents\": 1"}
EOF
cat > claims.json <<EOF
[{"kind":"claim","claim":"schema document prints the JSON Schema for transform output","source":"README.md","quote":"","input":"schema document","expected":"DocumentDefinition schema","probe_kind":"cli","argv":["$V","schema","document"],"cwd":"<guest-path>","expected_text":"\"model\": \"DocumentDefinition\""},
{"kind":"claim","claim":"mock test runs the connector without touching the network","source":"README.md","input":"snippet connector with static data","expected":"1 document posted to mocked Glean","probe_kind":"cli","argv":["$V","test","--phase","mock","--connector","complete:create_connector"],"cwd":"$P","env":{"WIKI_API_TOKEN":"placeholder-mock-token"},"expected_text":"\"documents\": 1"},
{"kind":"claim","claim":"every command takes --output json for a stable envelope","source":"README.md","input":"schema document --output json","expected":"JSON envelope with ok true","probe_kind":"cli","argv":["$V","schema","document","--output","json"],"cwd":"<guest-path>","expected_text":"\"ok\": true"}]
EOF
python3 -I -m json.tool probe.json >/dev/null && python3 -I -m json.tool claims.json >/dev/null && echo OK; cat probe.json
cd <guest-path> && V=<guest-path>; P=<guest-path>; cat > claims.json <<EOF
[{"kind":"claim","claim":"schema document prints the JSON Schema your transform must produce","source":"\`glean-idx schema document\` prints the JSON Schema your \`transform()\` has to","input":"schema document","expected":"DocumentDefinition JSON schema","probe_kind":"cli","argv":["$V","schema","document"],"cwd":"<guest-path>","expected_text":"\"model\": \"DocumentDefinition\""},
{"kind":"claim","claim":"mock test runs the connector without touching the network","source":"Test it without touching the network:","input":"README snippet connector with static data, mock phase","expected":"1 document posted to mocked Glean","probe_kind":"cli","argv":["$V","test","--phase","mock","--connector","complete:create_connector"],"cwd":"$P","env":{"WIKI_API_TOKEN":"placeholder-mock-token"},"expected_text":"\"documents\": 1"},
{"kind":"claim","claim":"every command takes --output json for a stable envelope","source":"Every command takes \`--output json\` for a stable envelope, \`--yes\` to skip","input":"schema document --output json","expected":"JSON envelope with ok true","probe_kind":"cli","argv":["$V","schema","document","--output","json"],"cwd":"<guest-path>","expected_text":"\"ok\": true"}]
EOF
python3 -I -m json.tool claims.json >/dev/null && echo VALID_JSON; grep -c 'glean-idx schema document' <guest-path>; grep -c 'Test it without touching the network:' <guest-path>; grep -c 'Every command takes `--output json` for a stable envelope, `--yes` to skip' <guest-path>
cd <guest-path> && V=<guest-path>; P=<guest-path>; cat > cycles.json <<EOF
{"detected":true,"checks":[{"claim":"the connector indexing job runs on a recurring schedule","source":"src/glean/indexing/deployment/config.py","quote":"default=\"0 2 * * *\", description=\"CronJob schedule (UTC cron expression).\"","schedule":"Kubernetes CronJob daily at 02:00 UTC (not triggered here; needs GKE/GCP secrets)","method":"run-once substitute: glean-idx test --phase mock runs the same connector full-mode fetch/transform/upload path once against mocked Glean","input":"snippet connector project, WIKI_API_TOKEN placeholder","expected":"one full-mode cycle posts 1 document to mocked Glean","kind":"cli","argv":["$V","test","--phase","mock","--connector","complete:create_connector"],"cwd":"$P","env":{"WIKI_API_TOKEN":"placeholder-mock-token"},"expected_text":"\"documents\": 1"}]}
EOF
cat > examples.json <<EOF
[{"kind":"example","claim":"deploy init generates Docker and Terraform files for a connector project","source":"README.md","input":"deploy init --cloud gcp --connector-factory create_connector into a fresh directory","expected":"generated_files list including glean_deployment.yaml","argv":["$V","deploy","init","--cloud","gcp","--connector-factory","create_connector","--output-dir","<guest-path>","-y"],"cwd":"<guest-path>","output_directory":"<guest-path>","expected_text":"glean_deployment.yaml"},
{"kind":"example","claim":"repository unit tests (documented mise run test task, expanded to uv run pytest)","source":"mise.toml [tasks.test]","input":"tests/","expected":"all unit tests pass; 4 optional cloud-plugin tests skipped","argv":["uv","run","pytest","-q","-p","no:logging","tests/"],"cwd":"<guest-path>","expected_text":"1017 passed, 4 skipped"}]
EOF
for f in cycles.json examples.json probe.json claims.json; do python3 -I -m json.tool $f >/dev/null && echo "$f ok"; done
cd <guest-path> && ls <guest-path> 2>&1; <env-var>=<guest-path> timeout 1500 python3 -I <guest-path> --validate --replay 2>&1 | tail -c 6000; echo; echo EXIT=${PIPESTATUS[0]}
cd <guest-path> && <env-var>=<guest-path> timeout 1500 python3 -I <guest-path> --validate --replay > <guest-path> 2>&1; echo EXIT=$?; python3 -I -c "
import json,sys
t=open('<guest-path>').read()
i=t.find('{')
d=json.loads(t[i:])
print({k:v for k,v in d.items() if not isinstance(v,(list,dict))})
for k,v in d.items():
    if isinstance(v,dict): print(k, {kk:vv for kk,vv in v.items() if not isinstance(vv,(list,dict))})
" 2>&1 | head -30
cd <guest-path> && cat > report.json <<'EOF'
{"outcome":"working","fault_owner":null,"operation":"Build and run the glean-indexing-sdk CLI from source, generate a connector project from the README quickstart snippet, and run its offline mock test (glean-idx test --phase mock), which fetches, transforms and posts one document to mocked Glean; also glean-idx schema document.","operation_scope":"Offline only. No live Glean instance, no Glean credentials, no GCP/Kubernetes. Python 3.12.3 with uv 0.13.5 (README requires Python >=3.10). Pinned commit 8b6a809814922ca77bf21f80d9b5bbbc6e1f457f (version 1.2.2). CDP endpoint http://<internal-ip>:9226/json/version returned HTTP 200 (Chrome/155), but no browser step was needed.","steps":["git init; git remote add origin https://github.com/gleanwork/glean-indexing-sdk; git fetch --depth 1 origin 8b6a809814922ca77bf21f80d9b5bbbc6e1f457f; git reset --mixed FETCH_HEAD","uv sync --locked --extra dev --extra test --extra lint --extra typing (mise.toml [tasks.setup])","uv run glean-idx --help; uv run glean-idx schema document","uv run pytest -q -p no:logging tests/ (mise.toml [tasks.test]): 1017 passed, 4 skipped","glean-idx deploy init --cloud gcp --connector-factory create_connector (README CLI section) to create a connector project","copy snippets/non_streaming/complete.py (README quickstart code) into the project as complete.py","glean-idx test --phase mock --connector complete:create_connector (README: test without touching the network) -> \"documents\": 1"],"deviations":["No mise installed; used the equivalent uv commands from mise.toml tasks setup and test directly.","glean-idx test requires a project folder containing glean_deployment.yaml; the README snippet was run from a project made with deploy init in <guest-path>, not in the snippet folder.","The README snippet's connector was used unmodified; WIKI_API_TOKEN was set to a placeholder because the snippet reads it at import time, and the mock phase never calls the source.","Cycle replay: the real trigger is a Kubernetes CronJob (default cron 0 2 * * *) running generated run.py, which needs GCP Secret Manager and a GKE cluster. It was not triggered. Substitute: one run of glean-idx test --phase mock, which runs the same full-mode fetch/transform/upload path against mocked Glean. This does not test the schedule, secrets, or real upload.","glean-idx doctor was run without credentials: it exits 3 with missing_credentials (GLEAN_SERVER_URL and GLEAN_INDEXING_API_TOKEN unset), matching the README's exit-code contract. No real credentials were available, so the live phases (integration/live) were not run.","Claim README 'Every command takes --output json' was tested on the schema command only.","The validator's example for deploy init uses a fresh output directory (<guest-path>) that the checker creates."],"stuck_step":null,"main_blocker":null,"plain_failure":null,"suggested_fix":null}
EOF
python3 -I -m json.tool report.json >/dev/null && echo REPORT_OK; ls -la <guest-path>; ls <guest-path> 2>&1 | tail -1