The Software Release Evidence Checklist: What a Green Pipeline Does Not Prove

A green pipeline is the most reassuring signal in software delivery and one of the least specific. It says that a set of jobs finished without a non-zero exit. It does not say which change they examined, whether they examined all of it, whether the results are recent, whether two of them disagree, or whether anyone with authority looked at the result and accepted it. Those are different claims, and a release decision depends on all of them.

This article gives release leads and compliance-minded QA teams a checklist built from eight properties that turn a pile of results into evidence. It includes a printable checklist table you can paste into a release ticket or run in a review meeting, an interactive matrix that shows how each kind of gap changes readiness, and a worked example. It does not promise compliance with any standard; it is a way of asking sharper questions before a release, and of being able to answer them afterwards.

Evidence is a result plus the identity of what it describes, the completeness of what it covers, and the time it was true.

The eight properties of release evidence

For each result you are about to rely on, ask whether it has these eight properties. The list is deliberately ordered from the most basic, where did this come from, to the most governance-specific, what stops the record being rewritten.

1. Source: where did this come from?

Every piece of evidence has a producer: a CI job, a scanner, a manual tester, an AI agent, a test-management tool. The producer matters because producers differ in how far they can be trusted and how easily they can be influenced by the change being judged. A report written by a protected pipeline is stronger evidence than a statement typed into a pull request description. Ask: who produced this, under whose control, and could the author of the change have altered it?

What to capture: the producer, the system it ran in, and the run or record identifier you can look up later. How it fails: an unlabelled screenshot, a pasted summary, or a badge with no link to the run behind it.

2. Target identity: what exactly does it describe?

A result describes a particular build of a particular commit in a particular environment. Without the identity, “tests passed” has no subject. Branch names are not identities, because branches move, and “the release candidate” is not an identity either, because candidates are rebuilt. The stable identifiers are the repository, the commit, the build or run, and the environment.

What to capture: repository, full commit identifier, build or run identifier and environment, recorded with the result. How it fails: a result cited for a later commit than the one it ran on, or staging evidence used for a production decision.

3. Completeness: did it cover everything it should have?

A report can be accurate and incomplete. A job that times out after running forty of four hundred tests produces a report in which everything passed. A scan that crashes may emit zero findings. Completeness is the property that the evidence covers the whole expected set, and it is the one most often assumed.

What to capture: the expected count or set against the actual one, and whether the producer reports that the run finished successfully. How it fails: partial runs, sharded runs where only one shard is submitted, and scans that did not actually complete.

4. Freshness: is it still true?

Evidence has a shelf life. A result from last week on a codebase that changed daily describes a different system. Freshness has two parts: evidence is superseded when a newer build becomes the target, and it is stale when it is simply too old, whatever the target. You need a written rule for each.

What to capture: the time the evidence was observed and the rule used to judge it. How it fails: a green run from before the last three merges, cited because nothing “really” changed.

5. Conflicts: does anything contradict it?

Real pipelines produce disagreement: a test fails then passes on retry, two environments differ, a scanner and a test suite tell different stories. A process that only looks at the latest result resolves every conflict in favour of the most recent run, which is how flaky failures and real regressions both disappear. Evidence is stronger when conflicts are visible and have to be resolved on purpose.

What to capture: every attempt, not only the latest, with a note of how a conflict was resolved. How it fails: the overwritten failure, where a rerun replaces the result and the history says the test always passed.

6. Exceptions and approvals: who accepted what, and why?

Releases sometimes go ahead despite an unmet condition. That can be the right call. What makes it governed rather than sloppy is a named person, a stated reason and a record of the condition they accepted, ideally with an expiry so the exception does not become permanent policy by neglect.

What to capture: the condition waived, the reason, the approver and the date. How it fails: a verbal “fine to ship” that no one can reconstruct, or an approval that links to nothing.

7. Decision: what was concluded, by what rule?

Evidence is not a decision. Someone or something has to apply a rule to the evidence and conclude. The strongest arrangement defines the rule before the run and applies it mechanically, so that the conclusion does not depend on who happens to be in the meeting. If a human decides, the decision should state the evidence it relied on.

What to capture: the conclusion, the rule or policy version applied, and the reasons. How it fails: the bar being set after the results are known.

8. Immutable history: can it be rewritten?

A record that can be edited is a draft. The final property is that history is append-only: when a failing decision is followed by a repaired one, both remain, and the order is visible. This is what lets a reviewer a year later see not only that you shipped, but how you got there.

What to capture: a retained, tamper-evident record of decisions and the evidence they relied on. How it fails: logs that expire, spreadsheets with edit rights for everyone, or a status field that is simply overwritten.

See how gaps change readiness

The matrix below lets you set the state of four typical pieces of evidence and watch the conclusion change. It uses a deliberately simplified rule to illustrate the principle, which is that evidence that is present but unusable is a problem of a different kind from evidence that is missing, and neither can be treated as a pass.

Try the presets. A stale integration run and a scan of another build both hold the change, even though every badge in a naive dashboard would be green. Marking one report missing leaves the decision incomplete, because there is nothing to clear yet. Only when every obligation is satisfied by complete, fresh evidence bound to the build does the outcome clear.

The printable checklist

The table below is designed to be pasted into a release ticket, a wiki page or a document, or printed from your browser. Run it in a fifteen-minute release review. The “Evidence to capture” column is what you will want to find later; the “Pass condition” column is what a reviewer checks now. Adapt the wording to your environment, and decide thresholds before the run.

Twenty-two lines is a lot for a small team. If you need to start smaller, begin with items 3, 4, 6, 10, 12 and 20. Those cover identity, completeness, freshness and history, which are where the largest silent gaps live. You can add the rest as the process matures.

Running it as a fifteen-minute release review

1. Minute 0 to 2: fix the target. State the release commit and environment aloud, and write them in the ticket. Every subsequent statement is judged against them.
2. Minute 2 to 6: evidence walk. For each required evidence type, the owner says the commit it names and shows the run. Anything that does not match is set aside, not argued about.
3. Minute 6 to 9: gaps. List what is missing, stale or conflicting. For each, decide whether to fetch new evidence or request an exception.
4. Minute 9 to 12: exceptions. For each exception, record the condition, reason, approver and expiry. If no one will put their name to it, it is not an exception; it is a blocker.
5. Minute 12 to 15: decide and record. The release owner states the conclusion and the reasons. Save the checklist with the ticket.

The ritual is simple on purpose. Its value is that the same eight questions are asked every time, so quality of release decisions stops depending on who attended.

A green pipeline that proves very little

Consider a release whose dashboard is entirely green. The unit job passed, the integration job passed, the scanner passed and the deploy job succeeded. Applying the eight properties changes the picture. The integration job is green because a retry masked two timeouts, so the record shows a single pass for tests that failed twice first (conflicts). The scanner badge belongs to the previous day’s commit, because a later merge changed a dependency (target identity and freshness). The unit report was produced by a workflow file that was itself edited in the same pull request (source). Nobody approved anything, because the pipeline was set to promote automatically on green (exceptions and decision), and the pipeline logs expire in thirty days (history).

Nothing in that release was necessarily wrong. The software may be fine. What is missing is the ability to say so with evidence. A release governed by the checklist would have held at the identity and conflict checks, fetched a scan for the right build, resolved the retried tests deliberately, protected the workflow file and recorded an approval. The extra effort is perhaps an hour. The difference is that the next incident review has something to read.

How much weight does each kind of producer deserve?

Not all evidence is equal, and a useful release process says so in advance. The ordering below is a reasoning aid rather than a rule: it reflects how much the author of a change can influence each producer, and how easily its output can be tied to an exact build.

• Protected CI test reports. Strong when the pipeline is protected, the report is complete and the build identity is recorded. This is the backbone of most gates. Its weakness is that it only reflects what the tests chose to check.
• Scanner reports. Strong on the class of problem the scanner understands, and silent on everything else. They need the same identity and completeness checks as test reports, plus confirmation that the scan actually finished.
• Test-management results. Valuable for manual and exploratory work and for traceability to requirements. Their weakness is that a human or a plugin recorded them, so they benefit from independent confirmation when they bear on a release, and from version pinning so they describe the test as it was.
• AI agent output. Useful as advice, proposals and explanations, and not as evidence that its own change works, because the agent is the author. Treat anything an agent asserts as a lead to verify.
• Human attestations. A named person saying “I checked” is meaningful when the statement says what was checked and when. It is weak when it is a bare approval.
• Dashboards and badges. Summaries of other evidence. Never cite them as evidence; follow them to the run behind them.

Writing this ranking down has an unexpected benefit. Teams discover that they have been relying on the weakest producers for the highest-stakes questions, such as using a dashboard badge as proof of a security scan, simply because it was the thing visible in the release meeting.

Keeping the record: what to store and for how long

The final property, immutable history, is the one that is easiest to neglect because it only pays off later. Practical guidance is modest. Store the decision record and the evidence it relied on somewhere that outlives your pipeline’s log retention, which is often as short as thirty or ninety days. Store enough to answer the questions an incident review asks: which commit, which evidence, which decision rule, who accepted which exception. And store it in a form that is append-only in practice, so that a later change adds a new entry rather than editing an old one.

Retention periods are a matter for your legal, security and regulatory advisers, and this article does not suggest one. What it does suggest is that the question be asked deliberately. A surprising number of teams discover during an incident that their only record of why a release went out is a chat thread, and that the thread has since been archived.

How this relates to standards and frameworks

Several public frameworks address aspects of this. The NIST Secure Software Development Framework describes practices for producing and verifying secure software, and SLSA defines levels of supply-chain assurance around how artifacts are built and attested. The DORA research programme studies delivery performance. These are useful references and they differ in scope: SLSA is about provenance of build artifacts, whereas this checklist is about the sufficiency of verification evidence for a release decision. Using a checklist like this one produces records that can support your own assessment against such frameworks. It does not, by itself, make a release compliant with any of them, and your auditor or assessor decides what is required.

Where TestWarden fits

Several of the eight properties are mechanical: matching evidence to a build, noticing stale or incomplete reports, retaining every attempt and refusing to rewrite history. The Warden automates those checks. It receives evidence from your CI and test tools with the repository, commit, build and environment, evaluates it against a verification contract and records a decision, HELD or CLEARED, with reasons. A repaired build gets a new decision and the original remains. Evidence about the wrong build, or that is stale or conflicting, does not count toward clearance.

The limits matter as much as the capability. TestWarden is publicly listed on Atlassian Marketplace; this article does not claim verified customer production deployment evidence. Customers use the default verification contract today; authoring your own, and approval or waiver options, are not built, so exceptions are still recorded in Jira by your own process. Evidence arrives as complete JUnit XML, SARIF 2.1.0 or a bounded JSON envelope. It is not a compliance certification, it carries no SOC 2 or ISO claim, and it cannot guarantee that software is free of defects. The Proof page lists what was shown, in which environment.

Frequently asked questions

Is a green pipeline not enough to release?

It is necessary in most teams and insufficient on its own. A green pipeline reports that jobs exited successfully, and says nothing about build identity, completeness, freshness, conflicts or authority unless those are separately checked.

How fresh should evidence be?

There is no universal number. Decide by how fast your code changes and how costly a stale pass would be. The important thing is that the rule is written down before the run. TestWarden’s default contract treats evidence older than 24 hours as stale, and newer builds supersede older evidence regardless of age.

What is the difference between superseded and stale?

Superseded evidence describes an older build than the current target and can never count for the newer one. Stale evidence is simply too old by the freshness rule, even if it matches the target.

Should flaky tests that pass on retry count as passes?

That is a policy decision, and the retry should leave a record either way. See flaky tests and release quality gates for the reasoning and a thought experiment.

Does this checklist make us compliant?

No. It produces a disciplined evidence trail that can support your own compliance process. Compliance is determined against a specific standard by your assessor.

Can we use this without TestWarden?

Yes. Everything on this page works with a ticket, a wiki and discipline. A tool helps when volume makes manual identity matching and history keeping unreliable.

Who should own the checklist?

The release owner runs it. The policy behind it, such as what counts as sufficient, is best owned by engineering or QA leadership and set before individual releases.