Quality Memory: the verification history TestWarden keeps

Quality Memory is the name for the verification history TestWarden keeps for your Jira site. It connects a requirement to the change that implemented it, the version of the test that checked it, the attempt that ran, the evidence that arrived, the failure or defect that followed, the decision made, and what was later observed. It is a record, not a model. Nothing in it is trained, tuned or pooled across customers. Its value is that it is complete, ordered, attributed and cannot be quietly edited.

That sounds modest, and it is meant to be. The reason it matters is what you can do with a trustworthy record that you cannot do without one. You can answer “why was this blocked and when was it released?” a year later. You can produce a dated audit trail without reconstructing it from CI logs and chat. You can see which reason codes block you most often. And, if you choose to enable it, you can let advisory intelligence read the same record to suggest what to check next, with every suggestion pointing back at the facts it came from. This article explains the model, the guarantees and the boundary between fact and interpretation.

The graph above has nine nodes in a chain. Select each one to see what it is, an invented example record and where the information comes from. Eight are authoritative facts. The last, Historical Intelligence, is marked as model interpretation. Below the graph, a five-step timeline shows how a failure in March can reach a suggestion in June without ever gaining the authority to decide anything.

The chain, node by node

TestWarden’s architecture centres on one question: given what the team intended to change, what actually changed, what history says is risky and what evidence now exists, what must be verified next, and why? The chain is the data structure that makes that question answerable.

Two design choices carry most of the weight. The first is that every node has a stable identity of its own. A Change Set is not “the PR called Fix refunds”; it is a repository, commit, build and environment. A Test Version is not “the refund test”; it is that test at that version. Because identities are exact, the edges between nodes are exact too, and the record can be queried and audited without heuristics. The second is that outcomes are recorded as outcomes. Whether a released change later caused an incident is a fact you add when it happens, linked to the change, rather than something a model infers later.

What is preserved

• Decisions. Each Warden decision: state (CLEARED, VERIFYING, HELD, NEEDS APPROVAL or INCOMPLETE), reason codes, next action, the Verification Contract version and the exact target. This answers “why was this blocked, and when was it released?” long after pipeline logs rotate.
• Evidence receipts. What each producer submitted: provider type, name, source system, run ID, format (JUnit XML, SARIF 2.1.0 or JSON) and the observations evaluated.
• Approvals. Who approved a NEEDS APPROVAL change, when and against which decision. A human override is a recorded event, not a silent merge.
• Test versions. Every edit to a test case, with archive and restore, so you can reconstruct what a test said on any date.
• Attempts. Each execution of a test, pinned to the version it ran, with results and linked defects. See version-pinned execution.

Append-only, by design

Decisions are never rewritten. If a change is HELD for failing evidence and a later build is cleared, there are two decisions: the block and the clearance. The same applies to submissions for older builds: they are kept as history and answered as superseded, but they do not alter the current target. In the product today, examples are labelled Contract v1 and Policy v1. Each decision records the contract version it was made under, so history stays interpretable when contracts change. Exact retries are idempotent, and changed or conflicting evidence is stored as a new receipt, never overwritten.

Why does this matter beyond tidiness? Because the credibility of a record comes from the fact that nobody, including you, can improve it afterwards. A history that can be edited is a narrative. One that can only be appended to is evidence. When an auditor, an incident reviewer or a customer asks what you knew and when, “here is the sequence, and it could not have been changed” is a qualitatively stronger answer than “here is what we believe happened”. The mechanics of an exact-build decision are walked through in a green check is not proof.

Facts and interpretation are kept apart

The most important distinction in the design is between authoritative facts and model interpretation. Facts are exact, server-authored records: this commit, this build, this attempt, this decision. Interpretation is a reading of those facts, such as “this test has failed after the last two changes to ledger code”. The first is true or false. The second is a hypothesis that may be useful.

This separation is enforced in how the product labels things. Only exact server-authored facts can carry a source-verified summary label. Counts, versions and citations in an advisory are checked against the canonical record, and claims of performed actions or authority are rejected. The safeguards exist because a language model is fluent whether or not it is right, and a fluent summary placed next to a decision can be mistaken for part of it. Keeping them visibly different protects the decision from the summary.

Tenant isolation and source attribution

• Per tenant. History is scoped to your installation on Atlassian Forge, in the app’s Forge storage. There is no cross-customer pooling and no cross-customer training. One customer’s failures do not inform another’s suggestions.
• Access follows Jira. Reading requires the right Jira role and a repository grant, and the Evidence APIs need a site-admin opt-in. Advisory reads verify current Jira and source access, so a user cannot read history about issues they cannot see.
• Attributed sources. Every receipt records the producer, source system and run. Every decision records the contract it was made under. Nothing arrives anonymously.
• Historical context stays historical. Imported context, such as a changed-files list or criteria as they read at the time, is retained as of that time and is not silently replaced by today’s version.

Attribution is not decoration. When evidence arrives from a pipeline, the decision can say which producer, from which run, said what. When something looks wrong later, you can follow it back. A record without attribution is just a pile of claims.

Why this is not continually retraining a model

“Memory” is a loaded word in AI products, and it often implies that a model learns from your data. That is not what happens here, and the difference is important for risk, for privacy and for trust.

1. No weights change. Quality Memory is a database, not a training set. Recording a decision does not alter any model. A suggestion reads the record at the time it is asked for.
2. It is explainable. A suggestion cites the attempts, receipts and decisions it used. A retrained model’s behaviour cannot be traced to specific records you can open.
3. It is deletable and correctable. Because history is structured records, retention and access follow the controls of your installation. Information diffused into model weights cannot be withdrawn.
4. It does not leak across tenants. There is no shared model improving on everyone’s failures, so there is nothing for one customer’s data to leak into.
5. It is auditable. The facts a suggestion rests on are the same facts an auditor can read. The model adds interpretation on top, never underneath.

The longer treatment of how an AI layer reads the memory, and what it may and may not do, is in Quality Memory and AI in software testing. The short version is that the record is the asset, and any model is a replaceable reader of it.

The intelligence that reads the record

TestWarden’s AI architecture is built on the same record. Its capabilities are Change Intelligence, CI Intelligence, impacted-test selection, test-gap intelligence, failure intelligence, release-risk intelligence and maintenance intelligence, plus a governed Quality Agent that can draft work for human approval. Each derived recommendation requires its own current opt-in control, administrators decide whether any of it runs, consent and a cost ceiling apply, and a model-backed invocation is limited to administrators.

Be clear about status. These capabilities are implemented in source and pass offline acceptance suites that use platform and SDK doubles rather than paid inference. Model-backed acceptance on the current source has not been completed, no numeric accuracy is published, and nothing here should be read as a claim of measured model quality or customer-live availability. The record, by contrast, is the core of the product and works without any model: the facts render whether or not anything interprets them. Release-risk output names missing or conflicting proof and the independent Warden state; it does not produce a numeric confidence score and does not clear anything.

A worked example across eight months

Imagine a small payments team using TestWarden. All names are invented. In March a change touches the ledger posting module. A test, T-31, fails in build 1042, the Warden records HELD with evidence_failed, and a Jira bug is linked. The failing observation, the receipt, the test version and the decision are each recorded with their identities, and the bug is attached to the Change Set.

In April the fix ships as build 1043. A new decision, CLEARED, is appended with fresh evidence. The HELD decision is untouched. After release, the team records the outcome: released, no incident. Nothing here required a model. It is bookkeeping done at the moment each thing happened, which is exactly why it can be trusted.

In June, a different engineer changes the refund path, which shares the ledger module. No requirement link ties the change to T-31, so a plain requirement-based lookup would not suggest it. If advisory impacted-test selection is enabled, it may offer T-31 as a candidate, with the rationale that the March attempt failed after a change to the same module, a citation to that attempt and decision, and a label saying the connection is inferred rather than confirmed. An engineer reads that and decides whether to run it. The Warden’s own decision for the new build still depends only on current evidence under the contract.

The example shows the whole design in miniature. Facts were recorded as they happened. A reader of those facts produced a useful, cited suggestion. A human judged it. The authority to clear or hold never moved. Without the record, the June suggestion would be unfounded; without the separation, the suggestion could quietly become a verdict.

suggestion:   Consider running T-31 "Ledger entry after retry"
basis:        inferred (shared module ledger/post.ts; no requirement link)
cites:        attempt T-31 v3 #2 (Failed, build 1042); decision HELD (evidence_failed, build 1042)
status:       advisory. Does not affect the Warden decision for this build.

What the record lets you answer

It is worth listing the questions in order of when they become answerable, because teams often expect value from history on the first day and are surprised when it takes months. Some questions are answerable immediately; others need a body of history.

• Day one. What decision applies to this build, and why? What evidence did it rest on? Which tests failed? Who approved an exception, if any?
• After a few release cycles. Which reason codes block us most often? Which tests are the most frequent subject of failures? How long between HELD and CLEARED?
• After many months. Which kinds of change tend to precede failures? Which areas have repeated defects? Which tests have never failed and may be candidates for review? These are the questions where advisory intelligence earns its keep, because a person scanning the history by hand will not do it consistently.

This is why the record is described as an asset rather than a report. A report is a view produced once. An asset accrues value as it grows, and what you pay for the value is discipline: capturing exact identities and outcomes at the time, every time. TestWarden makes that capture a by-product of normal work, because decisions, receipts and attempts are recorded as they occur.

What teams use it for

• Release review. Show what evidence supported each release candidate decision.
• Incident review. Find the decision for the build that shipped, the evidence behind it and who approved any exception.
• Audit requests. Produce a dated trail without reconstructing it from CI logs and chat. See audit-ready Jira test evidence.
• Recurring causes. Read your own history for the reason codes that block changes most often. That is analysis you do, with your own data.
• Choosing what to test. Where enabled, advisory candidate lists cite earlier failures, and an engineer decides whether to run them. See change impact analysis and test selection.

What Quality Memory is not

• Not learning in the model sense. It does not predict failures on its own, rank tests by itself or adapt the rules. Clearance is decided by deterministic policy against the contract.
• Not a guarantee. A clean history tells you what was shown, not that nothing is wrong. It is evidence, and evidence has limits.
• Not backfilled from other tools. Migration brings test cases, not executions. Your earlier results stay in the old tool, so archive them there. Quality Memory starts accumulating with the first decision and first attempt in TestWarden. Details are in the migration guides.
• Not a data lake for everything. It keeps what is needed to answer verification questions, not every log line your pipeline emits.

Frequently asked questions

To see how decisions are recorded, read a green check is not proof and decisions and approvals. For what has been demonstrated and what is still under acceptance, see Proof. If you want to evaluate the record on one real repository, contact us or see pricing.