Quality Memory: Give QA Agents Real Testing History Instead of Generic AI Context

Ask a general-purpose AI model how to test a payment retry and it will give you a competent, generic answer. Ask it which of your tests failed the last three times that module changed, or whether this particular failure has happened before, and it has nothing to say. It does not know your tests. It has no history to consult. Everything that would make advice specific to your system, the failures, the fixes, the flaky suites, the decisions people made and why, lives in your own records, if it lives anywhere.

That is the problem Quality Memory is meant to address. This article is a deeper treatment than the short introduction we published earlier: it covers what a testing history needs to contain to be useful, how it differs from training a model, how grounded AI should use it, what the privacy and cost boundaries are, and, carefully, which parts exist today and which are direction. If you manage QA or an AI platform and are weighing “AI for testing” claims, the distinctions here will help you ask better questions.

Memory is what happened, recorded so that it can be cited. Training is what a model absorbed, which cannot be cited. They are not the same thing.

What Quality Memory is, and what it is not

In TestWarden, Quality Memory is the tenant-specific, append-only history of your own quality work: evidence receipts with content digests, Quality Decisions with their reasons, test cases with version history, and version-pinned execution attempts. It belongs to one Jira site. It is a record you can read, and the platform’s documentation is explicit that it is a history, not a learning system, and that it is not used to train models.

It helps to be blunt about what the term does not mean, because “memory” in AI marketing is often used loosely.

• It is not a model that has learned from your tests. No model weights are continually trained on your data, and none are trained on everyone’s data. There is no cross-customer pooling.
• It is not a summary that the AI wrote about itself. The authoritative layer is the recorded facts, such as which attempt failed against which test version. Any AI-written text over it is a derived, advisory layer.
• It is not a promise of prediction. A history can inform a risk estimate. It does not make the estimate correct.
• It is not production-outcome monitoring. Continuous monitoring of what happens after release is not claimed.

Why history beats generic context, and why it is not training

There are three ways to make an AI system useful on a specific team’s testing. They are often blurred together, and they differ in what they cost, what they risk and what you can verify.

Quality Memory is the third approach. Its great advantage is auditability: when an explanation says a test failed in two of three executions, you can check that against the executions. Citations tie each statement to a recorded source, and a statement that cites a record that does not exist can be rejected mechanically. That is a property fine-tuning cannot give you, because a trained model cannot point at where it learned something.

The chain from requirement to historical intelligence

A useful history is a chain of linked records, not a pile of results. Each link answers a question a later reviewer, human or AI, will ask. The interactive chain below labels each link with its status. Select a link to read what it contains and how far it has been shown to work. The statuses use the same vocabulary as the rest of this site: proven in development, in source under acceptance, or direction that is not claimed.

Read the last link carefully. “Historical intelligence” is the most appealing and the least established part, and it is labelled as direction for a reason. The first six links are records. The seventh is what grounded analysis could do with them. The article continues with what each record needs to be genuinely useful.

What makes a testing record useful to a human or an agent

Identity: what exactly ran against what

A history is only as good as the identity of its entries. A failure that cannot be tied to a test version, a build and an environment is an anecdote. Version-pinned attempts, where each execution keeps a reference to the exact test version it ran, are the foundation, which is why we describe them separately in version-pinned test execution. Without pinning, editing a test rewrites the apparent meaning of every past result.

Completeness: failures stay

A history that drops or overwrites failures is useless for the questions that matter most, which are about failures. If a rerun replaces the failed attempt, the record says the test always passed, and any analysis built on it will conclude the test is stable. Append-only storage is not a nicety; it is what makes patterns detectable.

Provenance: where each fact came from

Each record should say who or what produced it, when, and from which source system. Provenance lets you weight evidence and lets a reviewer follow a claim back to its origin. It is also what makes citations checkable: a citation points at a record with a known source.

Decisions and reasons: the human layer

Results are not the whole story. A history that records that a release was held, why, who repaired it and what changed afterwards captures judgement that no report contains. TestWarden keeps Quality Decisions, including the reasons and the contract and policy versions used, and keeps the original decision when a repaired build is later cleared. This is the layer most likely to be missing in an ordinary test archive.

Links: requirements, defects and changes

Connections to Jira requirements, the Bugs raised from failed steps, and the changes under test turn a list of results into something a model can reason over. Requirement coverage derived from recorded executions is proven in development; richer links to source-control context are implemented in source and not fully accepted.

How grounded AI should use a history

There is a right and a wrong way for AI to sit on top of a testing history. The wrong way is to let it say what it likes. The right way constrains it in four steps that are worth recognizing in any product that claims to do this.

1. Retrieve narrowly. Build the request from the specific records the caller is authorized to see and that are relevant to the question, not from the whole archive.
2. Require citations. The answer must reference real records by identifier. A model answer with empty or invented citations is discarded.
3. Validate the shape. Accept only the expected structure. Extra keys, multiple objects, truncated responses and tool-call outputs are rejected.
4. Fail closed and stay advisory. If the answer is refused or invalid, show no AI text. If it is valid, present it next to the facts, never in place of them, and never let it set a decision.

TestWarden’s optional explanation action follows these rules. It is off by default, requires a site administrator to enable it, requires consent before any request is made, is subject to a spending cap, builds its prompt from references the caller may see, and fails closed. With AI disabled, not consented or over budget, the facts still render deterministically. In engineering acceptance dated 2026-10-27, a real bounded Claude explanation, run through Atlassian’s Forge LLM capability in a development installation, produced a correct answer about three executions of a test (two failures) across versions 4, 5 and 6, with citations matching real records. Earlier model answers that did not meet that bar were recorded as failed acceptance and were not counted.

Three questions a QA lead can ask, and what the history must contain to answer them

It is easier to judge a memory design by the questions it must answer than by its architecture diagram. Here are three, with the records each one needs. The scenarios are invented.

“Has this failure happened before?”

A payment-retry test fails in today’s run. To answer, the system needs prior attempts of the same test, each pinned to a test version, with the failing step and message, linked to the build each ran against. With those, a grounded answer can say: this test failed in two of its last three executions, at the same step, on test versions 4 and 6, and the version between them passed. Without version pinning, the answer collapses, because version 5 might have been a different test. Without failures retained, there is no “two of three” to cite.

“Why was last month’s release held?”

The answer lives in decisions, not results. The system needs the held decision, the reason codes, the evidence receipts it relied on, and the later repaired decision. A grounded answer can then explain that the integration report was incomplete for the first build, that the pipeline timed out, and that a new commit produced complete evidence and a cleared decision. The original hold being preserved is what makes the story recoverable; an overwritten status field would say only that the release is cleared.

“Which requirement is this change most likely to threaten?”

This one needs links: the change’s files, the requirements whose tests touch them, and the failure history of those tests. Some of those links are recorded facts, such as a requirement linked to a test in Jira, and some would be inferences, such as a file path resembling a test name. A responsible system distinguishes the two in its answer. This is the question where history helps least reliably, and where honesty about inference matters most. The change-impact article goes into it.

Notice what the three have in common. None requires the model to be clever about testing. Each requires the record to be good, and the model’s job is to read it accurately and say so. That is the reason to invest in the memory first and the intelligence second.

Building a testing history without any tool

The principles do not require TestWarden, and teams can improve their own position this week. Three changes carry most of the value. First, record the build identity next to every test result, so a failure is always attached to a commit. Second, stop overwriting: keep every attempt, and keep failed attempts as first-class records. Third, write down why releases were held or exceptions granted, in a place that is searchable and does not expire.

If you later add retrieval-based AI, you will be glad of those habits. The technique of giving a model retrieved documents with each request is well documented in the research literature, for example in the original retrieval-augmented generation paper, and its central lesson is the one this article keeps returning to: the quality of the answer is bounded by the quality of what is retrieved. For the testing-specific problem of non-deterministic results contaminating a history, Martin Fowler’s note on eradicating non-determinism in tests is a clear primer. A history full of flaky results is a poor memory, and cleaning the signal comes before building intelligence on top.

Privacy, isolation and cost boundaries

A testing history is sensitive. It can include names of internal systems, descriptions of failures and links to proprietary code. Any AI feature that reads it should be judged by its boundaries as much as by its answers.

• Tenant isolation. Records belong to one installation. TestWarden runs on Atlassian Forge, with data in Forge SQL and Object Store scoped to the installation. There is no cross-customer pooling. Same-owner two-site isolation has been checked in development, which is not commercial customer acceptance.
• Consent and control. Off by default, enabled by a site administrator, and consented before use, with a kill switch.
• Budget. Requests stop at a spending cap. In the development acceptance, a tiny fraction of an approved budget was used, and the unresolved portion of that ledger was conservatively recorded rather than treated as unbilled.
• Data egress. The submitted V1 declares no external egress and no third-party analytics. The optional explanation lane uses Atlassian’s Forge LLM capability rather than an external processor of our own.
• Deletion. On uninstall, Atlassian retains Forge-hosted data for its platform retention period of 28 days and then destroys it.

Cost deserves honesty too. Grounded AI is not free, and cost grows with how much history you retrieve and how often you ask. Budget caps are not a limitation to apologize for; they are what make an AI feature safe to leave on.

What exists today, precisely

Questions to ask any “AI for QA” vendor

1. Is the AI’s knowledge of my tests recorded history, retrieved per request, or weights trained on my data? Can I see which?
2. Can every claim in an answer be traced to a specific record?
3. What happens when the answer cites something that does not exist?
4. Is my data used to train anything, mine or anyone’s?
5. Can the AI change a decision, an approval or a stored state?
6. What does it cost per request, and what stops spend at a limit?
7. What is proven in a real installation, and what is only in a roadmap?

Frequently asked questions

Is TestWarden’s AI trained on our tests?

No. Quality Memory is a tenant-isolated history that grounded features retrieve. No model weights are trained on your data or pooled across customers.

What is the difference between Quality Memory and a test archive?

An archive stores results. Quality Memory is built to keep identity, failures, provenance, decisions and links together and append-only, so that questions about history can be answered and cited.

Can the AI clear a release using the history?

No. A deterministic policy owns clearance. AI may add explanatory text beside a decision and can never grant clearance, override a hold or manufacture evidence.

Is the AI feature available now?

The optional explanation lane is part of the development proof, not a generally available feature. The publicly listed Marketplace V1 has no user-facing AI; the optional Quality Memory and AI capabilities discussed here remain separate developer-preview work.

Can we import our old test history?

Test cases and steps can be imported from CSV exports. Historical execution results and attachments are not imported, so history begins with your first attempt after migration.

How is this different from change-based test selection?

Selection is one use of history: choosing which tests to run for a change. See change-based test selection for the evidence it needs and the limits of current capability.

What happens to the history if we uninstall?

Atlassian retains Forge-hosted data for 28 days and then destroys it. Export anything you need to keep before uninstalling.