Flaky Tests and Release Gates: Why Re-running Until Green Is Not Governance

Every team with a large test suite has met the flaky test: the one that fails on Tuesday, passes on a rerun and fails again on Friday. The instinct is to rerun. The pipeline goes green, the release proceeds, and everyone moves on. That instinct is understandable, usually harmless and occasionally catastrophic, and the reason it is occasionally catastrophic is the subject of this article.

It is written for CI and test-infrastructure engineers who own the gate between a green pipeline and a release. We will look at the arithmetic that makes retries so tempting, what a retry throws away, how freshness and rerun provenance should shape evidence, why a conflicting pair of attempts should be treated as a conflict rather than resolved in favour of the latest, and what safe decisions look like. A thought experiment with four gate policies applied to the same history makes the differences concrete. The recommendations are about process and apply to any gate; where we describe TestWarden’s behaviour, we say so and cite its decisions documentation.

Why rerunning is so tempting

Flakiness is a scale problem, and scale makes it feel unavoidable. If each of 200 tests has even a 0.5% chance of failing for reasons unrelated to the code, the chance that at least one fails in a given run is 1 minus 0.995 to the power of 200, which is about 63%. A suite that looks individually reliable fails most runs. That is the arithmetic behind the common experience that “CI is red again for no reason”, and it makes retry the obvious remedy. Published experience from large engineering organizations points the same way: Google’s testing blog has described how flaky tests are a persistent cost of scale, and CI vendors such as Bitrise document features for detecting and quarantining flaky tests, including the option to rerun failed tests so that a build can go green when they pass on retry.

None of that is wrong. Retries remove noise, and noise has a real cost: blocked pull requests, wasted investigation and a team that stops believing red builds. The issue is not that retries exist. The issue is what a retry-until-green gate concludes from them.

What a retry destroys

Consider a genuine defect, a race condition that makes a test fail on 30% of runs. This is not a flaky test; it is a flaky product, and the test is telling the truth. A gate that retries a failed test up to three times and passes if any attempt passes will let this build through whenever at least one of three attempts passes, which is 1 minus 0.3 cubed, or about 97.3% of the time. At a 10% failure rate the figure is 99.9%. The more intermittent the defect, the more reliably the retry policy hides it, because the policy is, in effect, a machine for finding a passing sample.

This is the central point. A retry converts an observation about the build, “it fails sometimes”, into a conclusion about the build, “it passes”, using only the fact that you sampled until you saw what you wanted. Information is destroyed at the moment the first failure is overwritten by the later pass. After that, nobody can tell a flaky test from a flaky product, and the history, which would have shown a pattern, records a clean pass.

Re-running until green is a sampling procedure that stops when it sees the answer it wanted.

Freshness and rerun provenance

Two properties of evidence are especially important when reruns are involved.

Freshness

A rerun produces new observations, and a gate must decide how they relate to old ones. Evidence has a shelf life: results older than your freshness window are stale, and results for an older commit than the current target are superseded. A rerun for an earlier commit cannot repair a later one. In TestWarden’s default contract, a complete current-build automated test run counts only if the evidence is no older than 24 hours, and evidence for an older build than the current target can never clear the change.

Rerun provenance

It should always be possible to tell, for any observation, which attempt produced it. In GitHub Actions the run identifier and attempt number are available, and carrying them with the evidence lets a gate distinguish the first attempt from a rerun. Without provenance, a gate sees only the final report and cannot know that it is the third try. Two further rules help. Rerun the whole job for a fresh build, because re-running only the gate resubmits an earlier attempt unchanged. And make exact retries idempotent while storing changed evidence as a new record, so that a network retry of a submission cannot be confused with a genuinely new observation, and a different result never overwrites an earlier one.

The same history, two views

The visual below shows three attempts of one test against the same commit: pass, then fail, then pass on a rerun with no code change. Toggle between the two views. The first shows only the latest result, which is what most dashboards and many gates use. The second shows the full attempt history. The attempts are invented, and how a rerun that disagrees with an earlier attempt is treated is a policy choice; the view shown is the stricter reading.

In the first view the build looks clean. In the second, the build has contradicted itself, and a contradiction is exactly what a release gate should not smooth over. The honest summary of three attempts against unchanged code with results pass, fail, pass is not “passed” and not “failed”. It is “unresolved”: something is nondeterministic, and we do not yet know whether it is the test, the environment or the product.

Conflicting attempts should hold, not average

There are three common ways to resolve conflicting attempts for the same build, and only one is safe by default.

• Latest wins. Simple and the most common. It turns every flaky failure into a pass if the last attempt happens to be green, and makes the outcome depend on when you stopped looking.
• Any pass wins. Equivalent to retry-until-green. It maximizes the pass rate and minimizes what the gate knows.
• Majority wins. Looks principled, and for a test failing 30% of the time it passes a defective build about 78% of the time over three attempts, since two or more passes out of three is likely. It also discards the minority observation, which is the one carrying the signal.
• Conflict holds. Passing and failing results for the same build are a conflict that must be resolved on purpose, by fresh complete evidence for a new build or by an authorized human decision with a stated reason.

The last option feels strict because it is. It accepts some false holds in exchange for never converting a contradiction into a pass. TestWarden’s default contract takes this line: a decision lists conflicting_evidence when passing and failing results contradict each other for the build, and that holds the change. It also holds on any failed, error or skipped observation, so a skipped test does not quietly count as a pass. Each hold names a next action, such as resolving the conflicting evidence and verifying again.

A deterministic thought experiment

Policies are easiest to compare on identical inputs, so here is a small model with no randomness. One test, one commit, and a history of attempts in order. We apply four gate policies and record the outcome. The sequences are invented and chosen to illustrate edge cases.

Read down the columns. “Any pass wins” never fails a build that ever passed, so it cannot distinguish the second row from the fourth. “Latest wins” gives different answers to the same set of observations depending on order, which means the gate’s decision is a function of the schedule, not only of the evidence. Majority is sensitive to how many retries are allowed. Only “conflict holds” is invariant to order, which is the property a deterministic gate needs: the same evidence should give the same decision.

That invariance is not an aesthetic preference. It is what makes a decision reproducible. A gate whose output changes when you shuffle the arrival order of identical observations is not evaluating the evidence; it is evaluating the evidence plus luck. In TestWarden’s default path, the same evidence under the same contract gives the same decision, and that is the property the conflict rule helps preserve.

What safe decisions look like

A gate that respects flakiness without being captured by it follows a handful of rules.

1. Keep every attempt. A rerun adds a record. It never replaces one. The history is the diagnostic instrument.
2. Label retries. Evidence should say it is attempt two of three. Gates and humans should be able to see how many tries it took.
3. Hold on conflict. Contradictory results for the same build are unresolved until a fresh build produces complete, consistent evidence or a named person decides otherwise and records why.
4. Fix by changing the build, not the verdict. The route from held to cleared is new evidence for a new commit, not a rerun of the old one. If the test was flaky for an environmental reason and you fixed the environment, that is a change worth a commit or a recorded exception.
5. Quarantine openly. If a known-flaky test must be removed from the gate to unblock work, that is a recorded exception with an owner and an expiry, and its results continue to be collected.
6. Measure flakiness as a first-class number. A flake rate per test, tracked over time, is how you tell a noisy test from a failing product and how you decide where to invest.

Retry, quarantine or fix?

Martin Fowler’s long-standing note on eradicating non-determinism in tests remains a clear guide to the usual causes, such as shared state, asynchronous behaviour, remote services and time, and is worth reading before deciding whether a given flaky test deserves a fix or a quarantine.

Diagnostic history is the real asset

Teams often treat flaky-test records as clutter. They are the opposite. A test that failed twice in the last ten attempts, at the same step, on three different commits, is telling you something specific, and you can only hear it if the failures were kept. A history that preserves attempts, pins each to the version of the test it ran, and links failures to the Bugs raised from them lets you separate three cases that look identical in a latest-result dashboard: a test that is noisy, a test that is correct about an intermittent product defect, and a test whose behaviour changed because someone edited it. See version-pinned test execution for the last of those.

Designing a retry policy that keeps the signal

If you are going to retry, and most teams will, design the policy so that it removes noise without removing information. A workable shape has four parts. Scope retries narrowly: retry at the level of the individual test or the infrastructure step that failed, not the entire suite, so the retry touches as little evidence as possible. Cap them low: one or two retries is enough to absorb a dropped connection, and every additional retry increases the chance of laundering a real intermittent defect, as the arithmetic above shows. Record every attempt with its number, so that the evidence for a build states that a test needed two tries. And distinguish the reason for the retry: a retry triggered by a recognized infrastructure signature, such as a timeout reaching a package registry, is different in kind from a retry triggered by an assertion failure, and the second should leave a much louder trace.

Then decide, in advance, what a retried-to-green result means for the gate. The answer can be generous, for example that a test which fails once and passes on retry marks the build as passed but opens a ticket and adds a point to that test’s flake counter. It can also be strict. The important thing is that the answer is written down, applied the same way to everyone and visible in the record. A policy you can state in a sentence is a policy a reviewer can check, and a policy that exists only in the retry configuration of one pipeline is a surprise waiting for the next incident review.

Telling the release owner the truth about a noisy build

The person who signs a release is rarely the person who understands the flaky suite. That gap is where retries do the most harm, because the release owner sees only “green”. A useful practice is to give them a single, plain statement alongside the status: how many tests needed retries in this build, which ones, how that compares with the usual rate, and whether any of them touch the area that changed. Something like “three tests passed only on retry, two are known-flaky and unrelated, one is in the payments module that this change modifies” changes a release conversation completely.

This is a good example of the broader principle that evidence is only as useful as its legibility. A gate that holds on conflict and explains why, with a next action, is communicating the truth about a noisy build in the only way that scales: as part of the decision itself, not as a footnote that people have to remember to look for.

What TestWarden does and does not claim

The Warden evaluates evidence for an exact repository, commit, build and environment. Failing, stale, conflicting or wrong-build evidence holds the change; evidence for an older build is kept as history and cannot clear a newer one; exact retries are idempotent and changed evidence becomes a new receipt; and decisions are append-only, so a repaired build gets a new decision while the original HELD stays on record. Real GitHub Actions runs against TestWarden’s repository demonstrated HELD then CLEARED in a development installation.

The limits: scenario tests cover stale, wrong-build, superseded and conflicting evidence, but the public proof records that those cases have not all been exercised live in an installation. Flaky-test and failure-pattern detection exist as advisory analyses in source and are not counted as operational capabilities. TestWarden is not a test runner or a retry manager; it judges the evidence your CI produces. Customers use the default contract, and customer-authored policies about how to treat retries are not available. TestWarden is publicly listed on Atlassian Marketplace; customer-authored retry policy remains outside the released V1.

Frequently asked questions

Should we ever retry failed tests automatically?

Yes, for transient infrastructure failures, provided each attempt is recorded and a fail-then-pass on the same build is visible as such. Do not let a retry replace the first observation.

Is a test that passes on retry a passing test?

It is an unresolved test. It may be flaky, it may be correct about an intermittent defect, or the environment may have recovered. A gate should treat it as a conflict until resolved.

Won’t holding on conflict slow us down?

It will hold some builds that were fine. That cost pushes teams to fix flakiness, which is the real remedy. If a hold is unacceptable for a specific known-flaky test, use an explicit, recorded exception.

How do I measure flakiness?

Track, per test, how often it produces different results on the same commit. That requires keeping every attempt with its commit identity.

What is the difference between a stale and a conflicting result?

Stale evidence is too old by your freshness rule. Conflicting evidence is recent and contradictory for the same build. They need different responses: rerun on current code, versus resolve the contradiction.

Does TestWarden detect flaky tests?

Not as an accepted capability today. It retains attempts and holds on conflict; flaky-test pattern analysis exists only as advisory source under acceptance.

Does this apply to non-GitHub CI?

The reasoning applies to any CI. Only GitHub Actions has been exercised with real runs in TestWarden’s proof so far.