Verification Contracts: Define What an Exact Software Change Must Prove

Every release process contains an unwritten contract. Somewhere in the heads of the people who ship software is a list of things that must be true before a change goes out: the tests ran, the scan was clean, nothing critical is open. The trouble with an unwritten contract is that it is renegotiated at every release, usually under time pressure, and applied differently by whoever is present.

A verification contract writes the list down, ties it to an exact software change, and lets a policy apply it the same way every time. This article is for staff engineers and platform architects who are designing or evaluating that mechanism. It explains the vocabulary, why evidence must be bound to an exact build, how fail-closed behaviour works, why contract and policy versions are recorded, and, importantly, what is not yet something customers can author. The product details come from TestWarden’s published decisions documentation; the reasoning applies to any system.

A small vocabulary

Precise terms keep the design discussion honest. These six are enough.

• Change set. The exact thing being judged, identified by repository, commit, build and environment. It is not “the branch” or “the release”. It is a specific build of a specific commit in a specific place.
• Evidence. A recorded observation produced by something: a test report, a scan, an agent run. Evidence has a producer, a time and a target identity.
• Obligation. A single requirement on evidence, such as “a complete automated test run exists for this build”.
• Verification contract. The set of obligations that apply to a change.
• Policy. The deterministic rules that evaluate evidence against the contract: what counts, how fresh it must be, how conflicts resolve, what holds the change.
• Decision. The recorded outcome for one change set, with the reasons, the contract and policy versions used, and the evidence relied upon.

The separation of contract from policy is subtle and useful. The contract says what must be proven. The policy says how proof is judged. Changing which obligations apply is a different act from changing how strict the freshness window is, and the two are best versioned and reviewed separately. It is also what makes a later question answerable: not only “what did we require?” but “by what rules did we judge it?”.

Why evidence is bound to an exact build

The central rule of a verification contract is that evidence counts only for the change set it names. This sounds obvious and is violated constantly in practice, by habit rather than by malice.

Consider why a looser rule fails. If evidence for “the repository” counts, a green run from last month satisfies today’s release. If evidence for “the branch” counts, any later push inherits the results of an earlier one. If evidence for “the same commit in a different environment” counts, a staging pass authorizes production. Each looser binding removes a way the evidence could be wrong from the discussion while leaving the possibility intact. Exactness is the only binding with no loophole: repository, commit, build, environment, all four.

The interactive example below shows two builds, A and B, with a set of evidence received. Switch the build under evaluation and watch which pieces count. The identifiers, including the contract and policy labels, are invented for illustration. Pay attention to the final state: evidence for build A cannot authorize build B, however complete it is for A.

Superseded versus stale

Two different rules sometimes get merged into “old evidence”, and a contract should keep them distinct. Superseded evidence describes an older build than the current target. Once a newer build becomes the target, evidence for the older one is history and cannot clear the new one, no matter how recent it is. Stale evidence matches the target but is simply too old by the freshness rule. In TestWarden’s default contract, a complete current-build automated test run counts only if its evidence is no older than 24 hours. The two failures call for different fixes: superseded evidence means run verification for the new build; stale evidence means run it again.

There is a further subtlety about the target. It only moves forward. When a newer commit becomes the verification target for a change, submissions for the older build are kept as history and answered as superseded, and they never clear anything. Without that monotonic rule an attacker, or a confused pipeline, could replay old evidence to move a change backwards to a state that once looked fine.

Fail closed: what happens when the answer is unclear

A verification system has to choose what to do when it cannot tell. There are two defaults, and they are not symmetric. Fail open treats uncertainty as a pass, so that the system never blocks delivery by mistake. Fail closed treats uncertainty as a hold, so that the system never clears a change by mistake. For a gate whose purpose is to prevent unproven changes from shipping, fail closed is the only coherent choice, and a contract should make each uncertain case explicit.

The default contract’s reason codes show how this plays out. They are a useful template for your own design because each corresponds to a distinct way proof can be unsatisfactory.

Two of these deserve comment. The default contract holds a build on any failed, error or skipped observation; a skipped test does not quietly count as a pass. And the last row shows a distinct outcome, incomplete: when the system cannot evaluate the change honestly because context, contract, policy or evidence is missing, malformed or too large, it does not pretend either way. Each hold names a next action, such as resolving failing evidence or resubmitting fresh evidence for this exact build, so that a hold is an instruction rather than a verdict.

Designing good obligations

If you are writing contracts, for your own system or in anticipation of tools that support them, a handful of principles separate useful obligations from ceremonial ones.

1. State the obligation as a property of evidence, not of people. “A complete automated run exists for this build” is checkable. “QA is happy” is not.
2. Require independent producers for the obligations that matter. If the author of the change can alter the producer, the obligation can be satisfied without being true. In TestWarden’s default contract, scanner and reviewer passes are recorded but cannot satisfy the test obligation, and agent evidence cannot satisfy the CI-run requirement.
3. Make completeness part of the obligation. “Tests passed” should mean “the expected set ran and passed”. Otherwise a short run clears.
4. Put a freshness rule in every obligation. Even a generous window is better than none.
5. Decide conflict resolution in advance. The worst time to decide how to treat a test that failed and then passed is during a release.
6. Keep the number small. Each obligation is a thing someone has to produce and maintain. A contract with forty obligations will be waived into meaninglessness.

An illustrative notation

It can help to sketch a contract before building one. The block below is purely a thinking aid. It is not a TestWarden file format and not something customers can author today; it is a way of making the six principles concrete.

contract: payments-service v3
target: repository + commit + build + environment
obligations:
  - id: automated-tests
    needs: complete test report from protected CI
    fresh: within 24h
    fails_on: any failed, error or skipped observation
  - id: security-scan
    needs: completed scan report for the same build
    fails_on: any finding above threshold
  - id: no-blocking-defects
    needs: authoritative defect context
    fails_on: any open blocking defect
policy: strict v2
  conflicts: hold
  missing: hold
  superseded: history only

Why contract and policy versions are recorded

A decision that does not record the rules that produced it is hard to trust later. Suppose a change was cleared in March under a freshness window of 24 hours, and in June the window was tightened to 4 hours. A reviewer looking at the March decision needs to know which rule applied, or the record will seem to violate current policy. Recording the contract and policy versions in every decision makes the history self-explanatory and lets you compare how rule changes would have altered past outcomes without rewriting them. In TestWarden today the default contract is labelled Contract v1 and Policy v1 in decisions.

The same idea applies on the testing side, which is why version-pinned execution matters. An attempt that records the exact version of the test it ran keeps its meaning after the test is edited. Version pinning at the contract level and at the test level are the same principle applied to different objects: a record should name the version of the thing that produced it so that it never silently changes what it says.

Edge cases that decide whether the design survives

Squash and rebase merges

Squash merges and rebases create new commits. Evidence bound to a pull request’s head commit does not automatically describe the commit that lands on the main branch. A contract design has to say which commit is the target at each stage, and either require fresh evidence for the merged commit or accept a documented equivalence. Pretending the identities are the same is a quiet source of wrong-build evidence.

Monorepos

A single commit may contain many independently releasable components. A contract tied to the repository and commit alone can be satisfied by evidence about one component while another is untested. Either scope obligations by component, or accept that the contract is for the whole commit and require evidence for all of it.

Matrix builds and multi-environment runs

Evidence from several platforms or shards is partial. Completeness has to be defined over the whole matrix. A single shard’s green report describes a slice.

Hotfixes and rollbacks

Emergencies are where contracts get waived. Decide beforehand whether a hotfix gets a reduced obligation set, who can approve it, and how that is recorded. A rollback to an earlier build is itself a change set with a target identity; the evidence that justified the old release may or may not still be fresh.

Reruns

Re-running a job for the same commit produces new observations for the same target. Whether they replace, supplement or conflict with the first attempt is a policy question. A conservative default is to keep every attempt and treat disagreement as a conflict to be resolved. See flaky tests and release gates.

Prior art: where the idea comes from

Binding claims to an exact artifact is not new, and it is worth knowing the neighbouring work so you can borrow vocabulary and avoid reinventing it. The supply-chain community has spent years on the problem of making statements about software that stay attached to the thing they describe. SLSA defines levels of assurance around build provenance: how an artifact was produced, from which source, by which builder. in-toto describes a framework for recording, as signed metadata, that each step of a supply chain happened as intended and was performed by the right party. Both are about attestations: statements about an artifact that a verifier can check.

A verification contract is the same shape of idea applied one level up, to the question of sufficiency rather than provenance. Provenance asks whether this artifact was built the way it claims. A contract asks whether the evidence about this change satisfies what we required. The two compose. A strong pipeline might obtain a provenance attestation for a build and a verification decision for the same change set, and a release policy could require both. They answer different questions, and neither replaces the other. Notably, this article makes no claim that TestWarden produces SLSA attestations or supports in-toto metadata; the reference is to show the family of ideas, not to assert compatibility.

One lesson carries over directly from that work: identity must be cryptographically or operationally unforgeable by the party being judged. In practice that means evidence is accepted from authenticated producers, the target identity is checked against records held by the platform rather than only against what the submission says, and the decision is recomputed by the verifier instead of trusted from the submitter. TestWarden’s approach to evidence reflects this in a limited way: the producer’s token is held by CI and never by an agent, submissions that try to carry a chosen state or approval are rejected, and the decision returned must match the repository, commit, build and environment that were submitted or the gate refuses to pass. Those are properties of the default path today. Stronger attribution for evidence collected from third-party test-management systems, such as signed source authority and independent attestation of upstream runs, is implemented in source and is awaiting live acceptance, and is not claimed as available.

Genuine limits today

It would be misleading to describe this as a general contract-authoring platform, so here is the actual state. Every TestWarden installation uses the default contract, labelled Contract v1 and Policy v1. Customer-authored contracts are not available, and neither are customer-selectable approval or waiver options. The current product has no customer-facing way to author a contract that requires a human approval or to grant one, so a NEEDS_APPROVAL state is not normally seen. Evidence arrives as complete JUnit XML, SARIF 2.1.0 or a bounded JSON envelope from CI, and there are no first-party adapters for specific scanners.

What is shown is the mechanism: exact-build binding, fail-closed holds, append-only history and decisions that cannot be set by the producer of the evidence. That behaviour was exercised with real CI runs in a Jira Cloud development installation, and the 2026-10-27 engineering acceptance reports a CLEARED decision for an exact source, independently confirmed by reading the native Check. TestWarden is publicly listed on Atlassian Marketplace; the evidence here was demonstrated in development, not in an independently verified customer deployment. The Proof page is the authoritative list of what has and has not been shown.

Questions to ask any system that claims contracts

1. What exactly is the target identity, and can evidence for a different build ever count?
2. What happens to evidence when a newer build becomes the target?
3. Is the freshness rule explicit, and can I see which evidence was judged stale?
4. What happens when two results contradict each other for the same build?
5. Can a producer assert its own success, or choose the outcome?
6. Does the decision record the contract and policy versions it used?
7. Is history append-only, and can a repaired decision erase the original?
8. Which parts can I author myself today, and which are fixed?

Frequently asked questions

Is a verification contract the same as a test plan?

No. A test plan lists what to test. A contract states what evidence a specific change must produce before it can proceed, which may include results from test plans along with scans and other checks.

Why separate the contract from the policy?

Because they change for different reasons and have different reviewers. The contract says what must be proven; the policy says how proof is judged. Versioning them separately makes decisions interpretable later.

What does fail closed cost?

It will sometimes hold a change that was actually fine, for example when a pipeline lost a report. That cost is visible and fixable; the cost of failing open is an unproven change shipping silently.

Can I write my own contract in TestWarden?

Not today. Every installation uses the default contract. Customer-authored contracts, waivers and approval options are not built.

Can an AI agent satisfy an obligation?

Agent evidence is recorded, and it cannot satisfy the requirement for an independent CI run. An agent can neither set a decision nor grant an approval.

What if evidence is for the right commit but the wrong environment?

It does not count. The environment is part of the target identity, so staging evidence cannot authorize production.

Is this a compliance framework?

No. It is a mechanism for making release decisions consistent and inspectable. It carries no certification claim.