Methodology
What a Coldpass result records, how a check is judged, and where its evidence stops.
How a trial runs
For each recorded attempt, an AI agent receives a public repository, its README and a task on a clean, disposable machine. The record identifies the source revision, task, model, operating system and test version. The agent’s report is checked separately against the documented behavior. A run can fail because of the project, missing prerequisites or our test setup; those causes are kept distinct.
What the six results mean
The result labels are Works, Works on some setups, Works with our fix, Didn’t work, Partly checked, Not tested. Works means a checker confirmed the recorded operation and the checked README claims passed on that setup. Works on some setups means recorded setups differ or some are incomplete. Works with our fix is reserved for a separately verified patch. Didn’t work means a current checked attempt failed on the tested setup. Partly checked means only part of the behavior was checked. Not tested means there is not enough current evidence for a result. A harness failure is our test problem, not a project verdict.
Checked README claims
A count such as “1 of 6 README claims checked” tells you how many recorded claims the checker evaluated for that setup. It is a count of checked claims, not a score for the README or a measure of everything the project can do. Unchecked claims remain unchecked. Each model, operating system, revision and task stands on its own.
Limits and what a result does not prove
A result describes one automated attempt and its attached evidence. It does not prove every feature works, guarantee compatibility with your environment, establish project quality, or certify security. A clean safety observation does not prove that software is safe. “Works” is not a guarantee; inspect the tested revision, task and evidence before relying on a result. Older checks may use different methods and are labeled accordingly.
Independence
Results are published whether they are favorable or unfavorable. Payment never buys a verdict, ranking or removal of a failure. A project can opt out without a fee; see opt out or request removal.