About Coldpass
Coldpass is a project by Taylor (TayHCode). It tests the path from a repo’s instructions to a useful result with an AI agent.
1. Start cold
A cheap agent gets the repo, its README and our questions on a fresh, disposable machine. Haiku is the default; Luna is the other tested model. We record the source revision, model, OS and test version. Local models and other agent clients are not established by these checks.
2. Check the work
A separate checker tries to confirm the claimed operation against documented behavior. Installing packages, starting a process or listing MCP tools alone does not establish useful behavior. New checks label evidence as README behavior checked, partly checked or no product behavior checked. Only the first can count as working under that test version.
3. Keep the limits visible
A demo proves that demo. A hosted MCP check does not prove local install. Needs a key, needs a browser and unsupported mean full use was not tested. A harness error is our test’s problem. Older results may use weaker checks; follow the exact run’s evidence and test version. Model and OS observations are separate, not a universal compatibility badge.
4. Observe safety and maintenance
Safety evidence describes static signals and, where collected, file, network and process behavior during the run. Missing observations stay unknown. This is not a security certification. Maintenance is a dated snapshot of public activity and replies, with its inputs and method linked from the result.
5. Prove a fix
A cold-proven fix needs a clean reproduction, upstream search, a patch, three failing cold runs before it and three passing cold runs after it on pinned revisions. A direct reproduction or filed fix alone is not cold-proven. Full records and available patch downloads are linked from the repo’s evidence page.
How we count
Targets: Count each selected public repository once, including blocked, older-only and not-yet-tested targets. In a review sample this is the selected sample, not the full catalog. Runs, models and platforms never add extra targets.
Works: Count a target once when at least one recorded OS/model setup has a checker-confirmed operation, explicit readme_checked strength and recorded README claims that all passed. Use the latest observation for each OS/model setup; exclude older checker evidence. Untested combinations stay unknown. If other recorded setups failed or remain incomplete, use Works on some setups. Each setup keeps its own revision, task and date; this is not a controlled comparison between models or platforms. Version, help, import and list-only operations are partly_checked; they do not establish useful product behavior. A proven-fix target belongs in the separate fix bucket.
Didn’t work: Count a target once when no current setup earns a checked pass or an operation-only partial result, at least one current setup has a confirmed repo failure, and no proven fix takes precedence. Missing credentials, unsupported setups and harness errors are unavailable evidence, not repo failures.
Works with our fix: Count each target once when the same verified fix data used by its card establishes a repository-owned failure with reproduction, upstream search, patch, three failing cold runs before and three passing cold runs after on pinned revisions. Duplicate fix records do not add targets. This proves the repaired example, not universal compatibility.
The remaining targets have Partly checked or Not tested. The build selects each target’s verdict once and shares its counts across home, browse, results and questions. Match the recorded revisions and checker versions in the exported run data; fix records and patches are linked from the changelog and repo evidence.
Inspect or repeat a result
Open a scorecard’s More details, then All the evidence for the task, recorded commands, checker, transcript and revisions. Read about the run data or read the test changes. Exports redact private data.
The public readtrial harness may lag the test version that produced a result. We do not claim that every current result can be reproduced by the public version. Match the recorded version before comparing a rerun.