{
  "run_id": "Coolhand-Labs--coolhand-python-claude-haiku-5-5-1793148580078383447",
  "checker": {
    "claims": [
      {
        "claim": "Streamed responses are automatically logged to Coolhand",
        "command": "<guest-path> examples/streaming_capture.py",
        "evidence": "PASS streaming_capture: 5 sync streams captured with token metadata\n",
        "expected": "streams captured with message id and token counts",
        "input": "sync SSE streams from anthropic and openai shapes via httpx",
        "kind": "claim",
        "outcome": "Real input produced the expected result: PASS streaming_capture: 5 sync streams captured with token metadata",
        "returncode": 0,
        "source": "Complete streamed response automatically logged to Coolhand!",
        "source_checked": true,
        "source_file": "README.md",
        "verdict": "yes"
      },
      {
        "claim": "Importing coolhand auto-initializes monitoring",
        "command": "<guest-path> examples/auto_monitor_example.py",
        "evidence": "Monitoring active: True\n\nCleaning up...\nData flushed and delivery attempted\n\n=======================================================\nAuto-monitor example completed!\n\nKey takeaways:\n1. Just 'import coolhand' can be enough to get started\n2. Auto-initialization detects AI/ML environments\n3. Zero configuration needed in many cases\n4. Pass silent=True/False and base_url to Coolhand() for environment-specific setups\n5. Environment variables control auto-initialization\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider",
        "expected": "monitoring reports active",
        "input": "import coolhand via auto_monitor_example.py",
        "kind": "claim",
        "outcome": "Real input produced the expected result: Monitoring active: True",
        "returncode": 0,
        "source": "Auto-initializes and starts monitoring",
        "source_checked": true,
        "source_file": "README.md",
        "verdict": "yes"
      }
    ],
    "cycle_checks": [
      {
        "claim": "Queued interactions are delivered by a background worker loop and drained on shutdown",
        "command": "<guest-path> examples/auto_monitor_example.py",
        "evidence": "Setting up demonstration environment...\n\nCoolhand Python SDK - Auto-Monitor Example\n=======================================================\nAuto-Initialization Example\n==================================================\n\u2713 Coolhand was automatically initialized!\n  Session ID: session_1793148889232\n  Monitoring enabled: True\n  Config source: Environment variables and auto-detection\n  Has API key: False\n\nZero-Configuration Usage\n------------------------------\nUsing Coolhand with zero configuration...\n\u2713 Submitted positive feedback\n\u2713 Submitted feedback with explanation\nMaking simulated OpenAI API request...\n\u2713 Request completed and automatically logged\nMaking simulated Anthropic API request...\n\u2713 Request completed and automatically logged\n\u2713 All requests automatically monitored and logged\n\nDifferent Setup Modes\n-------------------------\n\n1. Notebook Setup (interactive, verbose):\n   \u2713 silent: False\n\n2. Production Setup (silent):\n   \u2713 silent: True\n\n3. Self-Hosted Setup (custom base_url):\n   \u2713 base_url: https://coolhandlabs.com\n\nEnvironment Detection\n--------------------\nAuto-initialization activates when it detects:\n\u2713 AI API keys found: OPENAI_API_KEY\n\u25cb No AI/ML packages detected\n\u25cb Not in interactive session\n\nTo control auto-initialization:\n  - Set COOLHAND_AUTO_INIT=false to disable\n  - Set COOLHAND_ENABLED=true to force enable\n  - Set COOLHAND_API_KEY to enable with your API key\n\nMonitoring Context Manager\n-------------------------\nCurrent monitoring status: True\n\nTemporarily stopping and restarting monitoring:\n  Monitoring stopped \u2014 HTTP requests will pass through unlogged\n  Monitoring restored: True\n\nCollecting User Feedback\n-----------------------\n\u2713 Collected feedback on auto-monitoring experience\n\nFinal Status\n------------\nSession ID: session_1793148889232\nTotal interactions logged: 2\nMonitoring active: True\n\nCleaning up...\nData flushed and delivery attempted\n\n=======================================================\nAuto-monitor example completed!\n\nKey takeaways:\n1. Just 'import coolhand' can be enough to get started\n2. Auto-initialization detects AI/ML environments\n3. Zero configuration needed in many cases\n4. Pass silent=True/False and base_url to Coolhand() for environment-specific setups\n5. Environment variables control auto-initialization\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890106)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890110)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890114)\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring disabled\nINFO:coolhand:HTTP monitoring stopped\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring enabled\nINFO:coolhand:HTTP monitoring started\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete",
        "expected": "example reports data flushed and delivery attempted",
        "input": "auto_monitor_example.py with no COOLHAND_API_KEY",
        "kind": "cycle",
        "method": "run-once: auto_monitor_example.py enqueues interactions then calls shutdown/flush",
        "outcome": "Triggered one cycle by hand (run-once: auto_monitor_example.py enqueues interactions then calls shutdown/flush; normal trigger: queue-driven, no timer: worker blocks on the dispatch queue and drains on shutdown): Setting up demonstration environment...\n\nCoolhand Python SDK - Auto-Monitor Example\n=======================================================\nAuto-Initialization Example\n==================================================\n\u2713 Coolhand was automatically initialized!\n  Session ID: session_1793148889232\n  Monitoring enabled: True\n  Config source: Environment variables and auto-detection\n  Has API key: False\n\nZero-Configuration Usage\n------------------------------\nUsing Coolhand with zero configuration...\n\u2713 Submitted positive feedback\n\u2713 Submitted feedback with explanation\nMaking simulated OpenAI API request...\n\u2713 Request completed and automatically logged\nMaking simulated Anthropic API request...\n\u2713 Request completed and automatically logged\n\u2713 All requests automatically monitored and logged\n\nDifferent Setup Modes\n-------------------------\n\n1. Notebook Setup (interactive, verbose):\n   \u2713 silent: False\n\n2. Production Setup (silent):\n   \u2713 silent: True\n\n3. Self-Hosted Setup (custom base_url):\n   \u2713 base_url: https://coolhandlabs.com\n\nEnvironment Detection\n--------------------\nAuto-initialization activates when it detects:\n\u2713 AI API keys found: OPENAI_API_KEY\n\u25cb No AI/ML packages detected\n\u25cb Not in interactive session\n\nTo control auto-initialization:\n  - Set COOLHAND_AUTO_INIT=false to disable\n  - Set COOLHAND_ENABLED=true to force enable\n  - Set COOLHAND_API_KEY to enable with your API key\n\nMonitoring Context Manager\n-------------------------\nCurrent monitoring status: True\n\nTemporarily stopping and restarting monitoring:\n  Monitoring stopped \u2014 HTTP requests will pass through unlogged\n  Monitoring restored: True\n\nCollecting User Feedback\n-----------------------\n\u2713 Collected feedback on auto-monitoring experience\n\nFinal Status\n------------\nSession ID: session_1793148889232\nTotal interactions logged: 2\nMonitoring active: True\n\nCleaning up...\nData flushed and delivery attempted\n\n=======================================================\nAuto-monitor example completed!\n\nKey takeaways:\n1. Just 'import coolhand' can be enough to get started\n2. Auto-initialization detects AI/ML environments\n3. Zero configuration needed in many cases\n4. Pass silent=True/False and base_url to Coolhand() for environment-specific setups\n5. Environment variables control auto-initialization\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nNo matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nFailed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890106)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890110)\nINFO:coolhand:HTTP monitoring started\nINFO:coolhand:Coolhand initialized (session: session_1793148890114)\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring disabled\nINFO:coolhand:HTTP monitoring stopped\nINFO:coolhand.httpx_interceptor:Global HTTP monitoring enabled\nINFO:coolhand:HTTP monitoring started\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nWARNING:coolhand.feedback_service:No matching field provided. Feedback may not be linked to an LLM request. Consider providing one of: llm_request_log_id, llm_provider_unique_id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete",
        "returncode": 0,
        "schedule": "queue-driven, no timer: worker blocks on the dispatch queue and drains on shutdown",
        "source": "src/coolhand/client.py",
        "verdict": "yes"
      }
    ],
    "details": {
      "argv": [
        "<guest-path>",
        "examples/streaming_capture.py"
      ],
      "kind": "cli",
      "needs_browser": false,
      "needs_credentials": false,
      "returncode": 0,
      "scope": "product operation",
      "stderr": "",
      "stdout": "PASS anthropic via iter_bytes: id and token counts captured\nPASS anthropic via iter_lines: id and token counts captured\nPASS openai via iter_lines: id and token counts captured\nPASS openai via iter_bytes: id and token counts captured\nPASS openai stopping at [DONE]: id and token counts captured\nPASS streaming_capture: 5 sync streams captured with token metadata",
      "working": true
    },
    "evidence_strength": "readme_checked",
    "example_checks": [
      {
        "claim": "Offline streaming capture check passes without keys",
        "command": "<guest-path> examples/streaming_capture.py",
        "deferred": false,
        "evidence": "PASS anthropic via iter_bytes: id and token counts captured\nPASS anthropic via iter_lines: id and token counts captured\nPASS openai via iter_lines: id and token counts captured\nPASS openai via iter_bytes: id and token counts captured\nPASS openai stopping at [DONE]: id and token counts captured\nPASS streaming_capture: 5 sync streams captured with token metadata",
        "expected": "PASS lines and exit 0",
        "input": "streaming_capture.py",
        "kind": "example",
        "outcome": "command completed successfully",
        "returncode": 0,
        "source": "examples/README.md Streaming capture check",
        "verdict": "yes"
      },
      {
        "claim": "Zero-config monitoring example runs end to end",
        "command": "<guest-path> examples/auto_monitor_example.py",
        "deferred": false,
        "evidence": "id, original_output, or client_unique_id\nWARNING:coolhand.feedback_service:Failed to submit feedback: HTTP 401 - Unauthorized\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete\nINFO:coolhand.client:Shutting down Coolhand...\nINFO:coolhand.client:Coolhand shutdown complete",
        "expected": "example completes and reports delivery attempted",
        "input": "auto_monitor_example.py",
        "kind": "example",
        "outcome": "command completed successfully",
        "returncode": 0,
        "source": "examples/README.md table row auto_monitor_example.py",
        "verdict": "yes"
      },
      {
        "claim": "Repo test suite passes offline",
        "command": "make test",
        "deferred": false,
        "evidence": "omDramatiqThread::test_pydantic_ai_multiple_tasks\n  <guest-path> DeprecationWarning: The model 'claude-3-5-haiku-20241022' is deprecated and will reach end-of-life on February 19th, 2026.\n  Please migrate to a newer model. Visit https://docs.anthropic.com/en/docs/resources/model-deprecations for more information.\n    return cast(BetaMessage, await send(False))\n\n-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html\n================= 1413 passed, 3 skipped, 3 warnings in 45.97s =================",
        "expected": "pytest summary with passed tests",
        "input": "make test",
        "kind": "example",
        "outcome": "command completed successfully",
        "returncode": 0,
        "source": "Makefile test target",
        "verdict": "yes"
      }
    ],
    "status": "PASS"
  },
  "diagnosis": null
}