A green check is not proof: a HELD decision, step by step

A pipeline can be green while the change is not proven. The check may be for the previous commit, cover a subset of the tests, or come from a tool whose job is not to run your tests at all. A green check is a status that one system reports about one run. Proof is something else: evidence that is complete, attributable to this exact change, fresh, and evaluated against rules that were written before the result was known, by a party other than the one that produced it.

This walkthrough shows the difference using The Warden. It follows two illustrative changes: one with failing tests that sit beside a clean scan, and one whose “green” was really about a different build. In both you will see what a decision contains, which reason codes and next actions appear, why the repair creates a new decision instead of editing the old one, and what remains on the record afterwards. The repository, commit and test names are invented. The behaviour is how the product works.

The visual steps through the second example, a misattributed pass, from the green check to a new CLEARED decision, with the append-only history growing on the right. The full text of both examples follows, so the page makes sense without interacting.

Why a green check is a weak signal

Consider what a green check actually asserts. It says that a particular job in a particular workflow exited successfully. It does not say which commit it checked out, because caches, rebased branches and re-run jobs all conspire to make that less obvious than it sounds. It does not say how many tests ran, because a subset, a retried flake or a skipped suite all exit zero. It does not say who produced it, so it cannot distinguish a pipeline from an agent that wrote “checks passed” in a pull request description. And it says nothing about what was required in the first place, because the requirement lived in someone’s head or in a branch-protection list that nobody revisits.

• Wrong build. The check is real and green, but about the previous commit. The merge is of a commit nobody tested.
• Partial coverage. A smoke job passes, and its name suggests more than it proved.
• Wrong kind of evidence. A clean security scan is real and valuable. It is not a test run.
• Self-attestation. The party that made the change reports that it works.
• Silent failure. A job that never ran does not turn red. It simply never turns anything.

None of these is exotic, and none requires bad faith. They are the ordinary failure modes of status aggregation, and they get more frequent as the volume of changes rises. The remedy is not a better colour. It is to turn “is the status green?” into “is the evidence sufficient for this exact change?”, and to let something independent answer.

The change under review

A pull request is built by CI as build 1042 for commit a1b2c3d of acme/billing in environment staging. That tuple, repository, commit, build and environment, is the change identity. Everything the Warden says refers to it. The Warden evaluates it against the default Verification Contract, labelled Contract v1 and Policy v1 in the product today: a complete, current-build automated test run, with evidence no older than 24 hours. Any failed, errored or skipped observation prevents a pass. Scanner and reviewer passes are recorded but cannot satisfy the test obligation, and agent evidence cannot satisfy the CI-run requirement.

Notice what has already happened. The obligation was stated before any evidence arrived. It will not move to suit the result. That alone removes a whole class of argument, the kind that begins “well, the tests that matter passed.”

Example one, step 1: CI announces the run, state VERIFYING

Early in the pipeline, CI sends a run announcement (run_started). It is not evidence. It tells The Warden to expect some. The state becomes VERIFYING with reason verification_run_in_progress, and the default window is 60 minutes. If the window lapses without evidence, the state becomes HELD with verification_run_overdue, so a pipeline that dies silently does not look healthy. This is the answer to the silent-failure problem: absence is itself a state with a name.

Example one, step 2: failing JUnit with a clean scan, state HELD

The test job finishes and CI submits a JUnit XML report. Two tests failed: InvoiceRounding.carriesCents and TaxZone.fallsBackToDefault. Meanwhile a scanner submits a SARIF report with no findings, and a code-review agent submits a pass. The scanner and reviewer passes cannot satisfy the test obligation, and agent evidence cannot satisfy the CI-run requirement. The decision for build 1042 is:

Three things are worth noticing. First, the decision is specific: a state, a fixed reason code, a next action and named tests, not a vague red cross. Second, the scanner’s clean report is still on file as evidence, but it does not outweigh a failing test run. Third, the CI gate script warden-gate exits non-zero here. It exits 0 only for CLEARED on the exact repository, commit, build and environment it asked about, so a merge rule built on it cannot be satisfied by anything else.

Example one, step 3: the late submission for an old build

Someone re-runs the job for build 1041, which passes, and CI submits it. The verification target only moves forward, so this is recorded as history and answered with submissionSuperseded. It does not clear anything, and it does not undo the HELD on 1042. A green check from yesterday cannot launder today’s build. This rule matters most in busy repositories, where re-runs and late arrivals are normal, and where a naive “latest green wins” policy would let an old result clear a new commit.

Example one, step 4: the fix, a new build and a new decision

The developer or their agent fixes the rounding logic and pushes commit e4f5a6b. CI builds 1043, announces the run, submits a complete JUnit report with every observation passing, and the scanner reports clean again. The Warden produces a new decision for the new target: CLEARED. The earlier HELD decision for 1042 is never rewritten, so the record shows the block, the reason and the fix. warden-gate now exits 0 for build 1043. Re-running it with 1042 still fails.

Example two: when the green check is about the wrong build

The second example, the one in the visual, is harder to see in a normal pipeline. A pull request shows a green smoke check. It ran 12 of the suite’s tests, and, because of a caching mistake in the workflow, it checked out commit 7c9d1e0 (build 1041) rather than the pull request head a1b2c3d (build 1042). Nobody notices, because the status is attached to the pull request and the pull request is green.

The Warden’s obligation is a complete test run for build 1042. The only receipt on file is attributed to build 1041. Evidence about a different build is history, not proof, so the decision is HELD with reason no_matching_evidence and the next action resubmit_fresh_evidence_for_this_build. The merge gate reads the decision and blocks, regardless of the green smoke check. The repair is in the pipeline, not in the decision: the job must test the commit it claims to test. A new commit means a new build identity, CI announces the run, submits a complete report attributed to build 1043, and a new CLEARED decision is appended. The HELD for 1042 stays exactly as it was.

The anatomy of a decision

It is worth slowing down on what a Warden decision actually carries, because the structure is what makes it useful to three different readers at once. The pipeline needs a machine-readable answer it can gate on. The developer needs to know what to do next. The reviewer, or an auditor months later, needs to know what the answer rested on.

• State. One of CLEARED, VERIFYING, HELD, NEEDS APPROVAL or INCOMPLETE. Only CLEARED allows progression, and the gate’s exit code reflects exactly that.
• Reason codes. A fixed vocabulary, such as evidence_satisfied, evidence_failed, no_matching_evidence, only_stale_evidence, conflicting_evidence, verification_run_in_progress and verification_run_overdue. Fixed codes mean you can build dashboards and runbooks around them.
• Next action. A fixed instruction paired with the reason, for example resolve_failing_evidence or resubmit_fresh_evidence_for_this_build.
• Target. The repository, commit, build and environment the decision applies to. It is a statement about that build, not about a branch.
• Contract version. Which obligations the evidence was measured against.
• Evidence references. The receipts the decision rested on, with the producer, run and format of each.

Because the reason vocabulary is fixed, a team can write the response once. only_stale_evidence means re-run the required tests for this build because the matching evidence is older than 24 hours. conflicting_evidence means passing and failing results contradict each other for the build and someone has to resolve the contradiction, not average it. Compare that with a red cross on a pull request, which tells you only that something, somewhere, is unhappy.

Common objections

Teams that see this pattern for the first time raise the same handful of objections. They are reasonable, and the answers are specific rather than rhetorical.

“Branch protection already checks that required checks pass.” It does, and it is a valuable control. It checks that named statuses are green on the head commit. It does not check what each status covered, whether the evidence is complete, whether it is fresh, or what kind of evidence is required. It also cannot express “this obligation is satisfied only by a test run, not by a scan”, because to branch protection a status is a status.

“We have code review.” Review is valuable and complementary, and it is not a guarantee of correctness. Reviewers read diffs; they rarely re-derive whether a green check belonged to the commit in front of them. The Warden does not replace review. It makes sure the evidence a reviewer is implicitly relying on is attributable and sufficient.

“This will slow us down.” A decision is computed when evidence arrives, so the common path adds no human wait. What it removes is the slow path: the incident review where someone discovers the check was for the wrong build.

“Why not make CI itself the authority?” Because CI is a producer. The same system that runs the tests, holds the credentials and reports the status would also define what counts, which collapses the separation that makes the decision trustworthy. Keeping the authority separate also means you can change CI vendors without rewriting your definition of enough proof.

“Our agents write the tests too.” Agent-written tests are useful evidence about a change, and under this model they can be among the evidence submitted by the pipeline. They are not clearance, and the agent is not positioned to grade them. That argument is developed in why coding agents should not certify their own changes.

Independent authorization, in one sentence each

1. The author does not grade the change. Developers and agents produce code; they do not hold the credentials to submit evidence.
2. The producers do not decide. CI, scanners and reviewers report observations. A separate authority evaluates them.
3. The rules precede the result. The contract states what must be proven before the evidence exists.
4. The verdict is deterministic. The same evidence under the same contract gives the same state. AI can help explain it and never sets it.
5. The record is append-only. A later decision adds to history and never edits an earlier one.

These five properties are the working definition of independent software verification, and the reason the architecture is a control plane above your tools rather than a feature inside one of them. It matters more when code is produced by agents that also run the tests, which is the argument in why coding agents should not certify their own changes.

A checklist for your own pipeline

If you answered “only that a check passed”, “nothing” or “no” to several of these, your pipeline is relying on conventions the first busy week will break. None of them needs a new test runner to fix. They need an identity for the change, stated obligations and an independent evaluator.

Wiring it in without replacing anything

To try the pattern on one repository: opt in as a site admin, add a run announcement at the start of the test job, have CI submit the JUnit report with repository, commit, build and environment, and read the decisions next to your current merge rule. When they match your expectations, add warden-gate as a required step. The technical contract, including limits, is in CI evidence, the reading guide for reason codes and next actions is in decisions and approvals, and a broader checklist for what a release should be able to show is in the software release evidence checklist. See also the developer overview.

What to take away

• A decision always names an exact build, so a pass elsewhere cannot clear it.
• HELD tells you why (a fixed reason code), what to do (a next action) and which tests failed, up to ten.
• Different kinds of evidence satisfy different obligations. A scanner pass is not a test run.
• A pipeline that stays silent becomes VERIFYING and then HELD, not invisible.
• Decisions are appended. History is the audit trail.

Frequently asked questions

Next, read how The Warden works, see what has actually been demonstrated on Proof, or contact us to evaluate the pattern on one repository. Pricing describes the free tier for up to ten Jira users and the evaluation period.