{ "checker": { "claims": [ { "claim": "Streamed responses are automatically logged to Coolhand", "command": " examples/streaming_capture.py", "evidence": "PASS streaming_capture: 5 sync streams captured with token metadata\n", "expected": "streams captured with message id and token counts", "input": "sync SSE streams from anthropic and openai shapes via httpx", "kind": "claim", "outcome": "Real input produced the expected result: PASS streaming_capture: 5 sync streams captured with token metadata", "returncode": 0, "source": "Complete streamed response automatically logged to Coolhand!", "source_checked": true, "source_file": "README.md", "verdict": "yes" }, { "claim": "Importing coolhand auto-initializes monitoring", "command": " examples/auto_monitor_example.py", "evidence": "Monitoring active: True\n\nCleaning up...\nData flushed and delivery attempted\n\n=======================================================\nAuto-monitor example completed!\n\nKey takeaways:\n1. Just 'import coolhand' can be enough to get started\n2. Auto-initialization detects AI/ML environments\n3. Zero configuration needed in many cases\n4. Pass silent=True/False and base_url to Coolhand() for environment-specific setups\n5. Environment variables control auto-initialization\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider", "expected": "monitoring reports active", "input": "import coolhand via auto_monitor_example.py", "kind": "claim", "outcome": "Real input produced the expected result: Monitoring active: True", "returncode": 0, "source": "Auto-initializes and starts monitoring", "source_checked": true, "source_file": "README.md", "verdict": "yes" } ], "cycle_checks": [ { "claim": "Queued interactions are delivered by a background worker loop and drained on shutdown", "command": " examples/auto_monitor_example.py", "evidence": "Setting up demonstration environment...\n\nCoolhand Python SDK - Auto-Monitor Example\n=======================================================\nAuto-Initialization Example\n==================================================\n\u2713 Coolhand was automatically initialized!\n Session ID: session_1793148889232\n Monitoring enabled: True\n Config source: Environment variables and auto-detection\n Has API key: False\n\nZero-Configuration Usage\n------------------------------\nUsing Coolhand with zero configuration...\n\u2713 Submitted positive feedback\n\u2713 Submitted feedback with explanation\nMaking simulated OpenAI API request...\n\u2713 Request completed and automatically logged\nMaking simulated Anthropic API request...\n\u2713 Request completed and automatically logged\n\u2713 All requests automatically monitored and logged\n\nDifferent Setup Modes\n-------------------------\n\n1. Notebook Setup (interactive, verbose):\n \u2713 silent: False\n\n2. Production Setup (silent):\n \u2713 silent: True\n\n3. Self-Hosted Setup (custom base_url):\n \u2713 base_url: https://coolhandlabs.com\n\nEnvironment Detection\n--------------------\nAuto-initialization activates when it detects:\n\u2713 AI API keys found: OPENAI_API_KEY\n\u25cb No AI/ML packages detected\n\u25cb Not in interactive session\n\nTo control auto-initialization:\n - Set COOLHAND_AUTO_INIT=false to disable\n - Set COOLHAND_ENABLED=true to force enable\n - Set COOLHAND_API_KEY to enable with your API key\n\nMonitoring Context Manager\n-------------------------\nCurrent monitoring status: True\n\nTemporarily stopping and restarting monitoring:\n Monitoring stopped \u2014 HTTP requests will pass through unlogged\n Monitoring restored: True\n\nCollecting User Feedback\n-----------------------\n\u2713 Collected feedback on auto-monitoring experience\n\nFinal Status\n------------\nSession ID: session_1793148889232\nTotal interactions logged: 2\nMonitoring active: True\n\nCleaning up...\nData flushed and delivery attempted\n\n=======================================================\nAuto-monitor example completed!\n\nKey takeaways:\n1. Just 'import coolhand' can be enough to get started\n2. Auto-initialization detects AI/ML environments\n3. Zero configuration needed in many cases\n4. Pass silent=True/False and base_url to Coolhand() for environment-specific setups\n5. Environment variables control auto-initialization\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890106)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890110)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890114)\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring disabled\nINFO:coolhand:HTTP monitoring stopped\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring enabled\nINFO:coolhand:HTTP monitoring started\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete", "expected": "example reports data flushed and delivery attempted", "input": "auto_monitor_example.py with no COOLHAND_API_KEY", "kind": "cycle", "method": "run-once: auto_monitor_example.py enqueues interactions then calls shutdown/flush", "outcome": "Triggered one cycle by hand (run-once: auto_monitor_example.py enqueues interactions then calls shutdown/flush; normal trigger: queue-driven, no timer: worker blocks on the dispatch queue and drains on shutdown): Setting up demonstration environment...\n\nCoolhand Python SDK - Auto-Monitor Example\n=======================================================\nAuto-Initialization Example\n==================================================\n\u2713 Coolhand was automatically initialized!\n Session ID: session_1793148889232\n Monitoring enabled: True\n Config source: Environment variables and auto-detection\n Has API key: False\n\nZero-Configuration Usage\n------------------------------\nUsing Coolhand with zero configuration...\n\u2713 Submitted positive feedback\n\u2713 Submitted feedback with explanation\nMaking simulated OpenAI API request...\n\u2713 Request completed and automatically logged\nMaking simulated Anthropic API request...\n\u2713 Request completed and automatically logged\n\u2713 All requests automatically monitored and logged\n\nDifferent Setup Modes\n-------------------------\n\n1. Notebook Setup (interactive, verbose):\n \u2713 silent: False\n\n2. Production Setup (silent):\n \u2713 silent: True\n\n3. Self-Hosted Setup (custom base_url):\n \u2713 base_url: https://coolhandlabs.com\n\nEnvironment Detection\n--------------------\nAuto-initialization activates when it detects:\n\u2713 AI API keys found: OPENAI_API_KEY\n\u25cb No AI/ML packages detected\n\u25cb Not in interactive session\n\nTo control auto-initialization:\n - Set COOLHAND_AUTO_INIT=false to disable\n - Set COOLHAND_ENABLED=true to force enable\n - Set COOLHAND_API_KEY to enable with your API key\n\nMonitoring Context Manager\n-------------------------\nCurrent monitoring status: True\n\nTemporarily stopping and restarting monitoring:\n Monitoring stopped \u2014 HTTP requests will pass through unlogged\n Monitoring restored: True\n\nCollecting User Feedback\n-----------------------\n\u2713 Collected feedback on auto-monitoring experience\n\nFinal Status\n------------\nSession ID: session_1793148889232\nTotal interactions logged: 2\nMonitoring active: True\n\nCleaning up...\nData flushed and delivery attempted\n\n=======================================================\nAuto-monitor example completed!\n\nKey takeaways:\n1. Just 'import coolhand' can be enough to get started\n2. Auto-initialization detects AI/ML environments\n3. Zero configuration needed in many cases\n4. Pass silent=True/False and base_url to Coolhand() for environment-specific setups\n5. Environment variables control auto-initialization\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890106)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890110)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890114)\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring disabled\nINFO:coolhand:HTTP monitoring stopped\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring enabled\nINFO:coolhand:HTTP monitoring started\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete", "returncode": 0, "schedule": "queue-driven, no timer: worker blocks on the dispatch queue and drains on shutdown", "source": "src/coolhand/client.py", "verdict": "yes" } ], "details": { "argv": [ "", "examples/streaming_capture.py" ], "kind": "cli", "needs_browser": false, "needs_credentials": false, "returncode": 0, "scope": "product operation", "stderr": "", "stdout": "PASS anthropic via iter_bytes: id and token counts captured\nPASS anthropic via iter_lines: id and token counts captured\nPASS openai via iter_lines: id and token counts captured\nPASS openai via iter_bytes: id and token counts captured\nPASS openai stopping at [DONE]: id and token counts captured\nPASS streaming_capture: 5 sync streams captured with token metadata", "working": true }, "evidence_strength": "readme_checked", "example_checks": [ { "claim": "Offline streaming capture check passes without keys", "command": " examples/streaming_capture.py", "deferred": false, "evidence": "PASS anthropic via iter_bytes: id and token counts captured\nPASS anthropic via iter_lines: id and token counts captured\nPASS openai via iter_lines: id and token counts captured\nPASS openai via iter_bytes: id and token counts captured\nPASS openai stopping at [DONE]: id and token counts captured\nPASS streaming_capture: 5 sync streams captured with token metadata", "expected": "PASS lines and exit 0", "input": "streaming_capture.py", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "examples/README.md Streaming capture check", "verdict": "yes" }, { "claim": "Zero-config monitoring example runs end to end", "command": " examples/auto_monitor_example.py", "deferred": false, "evidence": "id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete", "expected": "example completes and reports delivery attempted", "input": "auto_monitor_example.py", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "examples/README.md table row auto_monitor_example.py", "verdict": "yes" }, { "claim": "Repo test suite passes offline", "command": "make test", "deferred": false, "evidence": "omDramatiqThread::test_pydantic_ai_multiple_tasks\n DeprecationWarning: The model 'claude-3-5-haiku-20241022' is deprecated and will reach end-of-life on February 19th, 2026.\n Please migrate to a newer model. Visit https://docs.anthropic.com/en/docs/resources/model-deprecations for more information.\n return cast(BetaMessage, await send(False))\n\n-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html\n================= 1413 passed, 3 skipped, 3 warnings in 45.97s =================", "expected": "pytest summary with passed tests", "input": "make test", "kind": "example", "outcome": "command completed successfully", "returncode": 0, "source": "Makefile test target", "verdict": "yes" } ], "status": "PASS" }, "deviations": [ "README says 'pip install coolhand'; installed the snapshot source instead to test the pinned code.", "The library has no console script or CLI; the documented runnable entry points are the examples/ scripts, which were used as the probe entry points.", "First run of examples/streaming_capture.py failed with ModuleNotFoundError: httpx (httpx is only in the 'test' extra); fixed by running the README-documented make install.", "HelpCenterDocs and Coolhand API calls were not exercised: no product CLI and no COOLHAND_API_KEY. Feedback during auto_monitor_example returns HTTP 401 without a key, which is expected for an unkeyed run.", "Git pin not applied: the workspace has no git metadata and I did not run the git init/fetch steps, so the snapshot was used as delivered.", "CDP endpoint: GET http://:9226/json/version returned HTTP 200 with Chrome/155 JSON; no browser was needed by this SDK." ], "fault_owner": null, "main_blocker": null, "operation": "Run the coolhand Python SDK's offline monitoring examples and repo test suite: HTTP stream capture (examples/streaming_capture.py) and zero-config auto-monitoring (examples/auto_monitor_example.py)", "operation_scope": "Local source snapshot at pinned tree, Python 3.12.3, Linux guest, no COOLHAND_API_KEY and no provider keys. Covers SDK interception of httpx streams and auto-init, not live delivery to Coolhand servers or provider calls.", "outcome": "working", "plain_failure": null, "steps": [ "python3 -m venv && pip install (coolhand 0.11.1 installed from source)", "make install (uv sync --all-extras, creates .venv on Python 3.12.3)", "uv run python examples/streaming_capture.py (PASS x5, exit 0)", "examples/auto_monitor_example.py (completed, exit 0)", "make test (1413 passed, 3 skipped)", "= python3 -I --validate --replay (valid=true)" ], "stuck_step": null, "suggested_fix": null }