mossadnik/pyspark-inspect

Review signals: 50 known vulnerability.

Reviewed as expected for a test run: attempted writes that failed, so nothing was written; resolver address lookups that send no data.

OSV reports a known vulnerability in this dependency version. 11 distinct locations. 50 findings.

OSV scanned supported lockfiles; unpinned dependencies and unsupported formats are outside this check.

A short run cannot establish that code is safe.

Connections show IP addresses; proxies can hide the final hostname.

Tracing covers dispatched commands and their children, not the whole OS; guest root can interfere with tracing.

Full safety evidence →

• inspect a PySpark dataframe query plan: Probe runs agent-written code rather than a documented product command.

  Source: It converts analyzed Catalyst plans into Python data structures that can be queried programmatically.

  Input: Spark dataframe with a+b column, matching Basic Usage

  Expected: parsed plan containing Project

  Command: <guest-path> .coldpass/operation.py

  Observed: Probe runs agent-written code rather than a documented product command.

Examples

• README Basic Usage dataframe inspection: command output contains a crash diagnostic

  Source: README.md, Basic Usage

  Input: The README Spark dataframe expression with a+b column

  Expected: Parsed Catalyst plan

  Command: <guest-path> .coldpass/operation.py

  Observed: s/workspace/pyspark_inspect/api.py", line 9, in inspect_dataframe return parse_plan(load_catalyst_plan(get_dataframe_plan(df))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<guest-path>", line 246, in parse_plan children = [parse_plan(c) for c in plan.children] ^^^^^^^^^^^^^ File "<guest-path>", line 249, in parse_plan raise ValueError(f'Unsupported plan: {plan.class_name}') ValueError: Unsupported plan: org.apache.spark.sql.execution.LogicalRDD

Inspect a PySpark dataframe query plan using the README Basic Usage example

pwd && ls -la && sed -n '1,240p' README.md && python3 --version && id -u
cat pyproject.toml && rg --files && rg -n "def inspect|inspect_dataframe|cron|schedule|__main__" pyspark_inspect tests
find pyspark_inspect tests -type f -maxdepth 4 -print; sed -n '1,240p' pyspark_inspect/__init__.py; sed -n '1,240p' pyspark_inspect/inspect.py
python3 -m venv .venv && .venv/bin/python -m pip install -e .
sed -n '1,240p' pyspark_inspect/api.py; sed -n '1,200p' pyspark_inspect/plan.py
JAVA_TOOL_OPTIONS='-XX:ActiveProcessorCount=2' .venv/bin/python .coldpass/operation.py
mkdir -p .coldpass; cat > .coldpass/operation.py <<'PY'
from pyspark.sql import SparkSession, functions as F
from pyspark_inspect import inspect_dataframe
spark = SparkSession.builder.master('local[1]').appName('coldpass').config('spark.ui.enabled', 'false').getOrCreate()
spark.sparkContext.setLogLevel('ERROR')
df = spark.createDataFrame([[1, 2]], ['a', 'b']).withColumn('c', F.col('a') + F.col('b'))
plan = inspect_dataframe(df)
print(type(plan).__name__)
print(plan)
spark.stop()
PY
JAVA_TOOL_OPTIONS='-XX:ActiveProcessorCount=2' .venv/bin/python .coldpass/operation.py
java -version; sudo -n apt-get -o DPkg::Lock::Timeout=300 update && sudo -n apt-get -o DPkg::Lock::Timeout=300 install -y default-jre-headless
JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 JAVA_TOOL_OPTIONS='-XX:ActiveProcessorCount=2' .venv/bin/python .coldpass/operation.py
 .venv/bin/python -m pip install 'pyspark==3.5.7'
JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 JAVA_TOOL_OPTIONS='-XX:ActiveProcessorCount=2' .venv/bin/python .coldpass/operation.py
sed -n '1,220p' pyspark_inspect/parser/catalyst.py; sed -n '1,120p' tests/conftest.py
cat > .coldpass/probe.json <<'JSON'
{"kind":"cli","argv":["<guest-path>",".coldpass/operation.py"],"cwd":"<guest-path>","env":{"JAVA_HOME":"/usr/lib/jvm/java-21-openjdk-amd64","JAVA_TOOL_OPTIONS":"-XX:ActiveProcessorCount=2"},"expected_returncode":1,"expected_text":"Unsupported plan: org.apache.spark.sql.execution.LogicalRDD"}
JSON
cat > .coldpass/claims.json <<'JSON'
[{"kind":"claim","claim":"inspect a PySpark dataframe query plan","source":"It converts analyzed Catalyst plans into Python data structures that can be queried programmatically.","input":"Spark dataframe with a+b column, matching Basic Usage","expected":"parsed plan containing Project","probe_kind":"cli","argv":["<guest-path>",".coldpass/operation.py"],"cwd":"<guest-path>","env":{"JAVA_HOME":"/usr/lib/jvm/java-21-openjdk-amd64","JAVA_TOOL_OPTIONS":"-XX:ActiveProcessorCount=2"},"expected_returncode":1,"expected_text":"Unsupported plan: org.apache.spark.sql.execution.LogicalRDD"}]
JSON
cat > .coldpass/cycles.json <<'JSON'
{"detected":false,"reason":"Inspected README, pyproject.toml, and package entry points; the library exposes inspect_dataframe and has no scheduling, loop, webhook, or cycle behavior.","checks":[]}
JSON
cat > .coldpass/examples.json <<'JSON'
[{"kind":"example","claim":"README Basic Usage dataframe inspection","source":"README.md, Basic Usage","input":"The README Spark dataframe expression with a+b column","expected":"Parsed Catalyst plan","argv":["<guest-path>",".coldpass/operation.py"],"cwd":"<guest-path>","env":{"JAVA_HOME":"/usr/lib/jvm/java-21-openjdk-amd64","JAVA_TOOL_OPTIONS":"-XX:ActiveProcessorCount=2"},"expected_returncode":1,"expected_text":"Unsupported plan: org.apache.spark.sql.execution.LogicalRDD"}]
JSON

cat > .coldpass/report.json <<'JSON'
{"outcome":"blocked","fault_owner":"repo","operation":"Inspect a PySpark dataframe query plan using the README Basic Usage example","operation_scope":"Python 3.12.3, PySpark 3.5.7, Java 21; editable install of pyspark-inspect","steps":["python3 -m venv .venv",".venv/bin/python -m pip install -e .","sudo -n apt-get -o DPkg::Lock::Timeout=300 update","sudo -n apt-get -o DPkg::Lock::Timeout=300 install -y default-jre-headless",".venv/bin/python -m pip install pyspark==3.5.7","JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 JAVA_TOOL_OPTIONS='-XX:ActiveProcessorCount=2' .venv/bin/python .coldpass/operation.py"],"deviations":["Initial editable install selected PySpark 4.2.1; the sample failed with Unsupported plan: org.apache.spark.sql.execution.LogicalRDD. Repeated with README-tested Spark 3.5.7 and observed the same error."],"stuck_step":"The documented Spark dataframe example calls inspect_dataframe, whose parser rejects the LogicalRDD created by spark.createDataFrame.","main_blocker":"Unsupported plan: org.apache.spark.sql.execution.LogicalRDD","plain_failure":"The README sample cannot inspect its dataframe: the parser stops with 'Unsupported plan: org.apache.spark.sql.execution.LogicalRDD' under both PySpark 4.2.1 and the documented tested Spark 3.5.7.","suggested_fix":"Add parser support for Spark's LogicalRDD nodes, then rerun the README dataframe example on Spark 3.5."}
JSON
<env-var>=<guest-path> python3 -I <guest-path> --validate --replay
cat > .coldpass/probe.json <<'JSON'
{"kind":"cli","argv":[".venv/bin/spark-submit","--master","local[1]","--conf","spark.ui.enabled=false","--conf","spark.driver.extraJavaOptions=-XX:ActiveProcessorCount=2",".coldpass/operation.py"],"cwd":"<guest-path>","env":{"JAVA_HOME":"/usr/lib/jvm/java-21-openjdk-amd64"},"expected_returncode":1,"expected_text":"Unsupported plan: org.apache.spark.sql.execution.LogicalRDD"}
JSON

<env-var>=<guest-path> python3 -I <guest-path> --validate --replay