Change-Based Test Selection: How to Prioritize Regression Tests When AI Writes More Code

Running every test on every change is the safest strategy and, past a certain size, an impossible one. Suites grow, pipelines slow, and the pressure to run only the tests that matter becomes irresistible. AI-assisted development sharpens the problem from both sides: it produces more changes, which multiplies the retesting bill, and it produces changes whose side effects are less well understood by the people shipping them.

Choosing a subset of tests is therefore a risk decision dressed as an optimization. This article is for engineering and test-automation leads who have to make that decision. It explains the main selection methods, what evidence each one needs, where each fails, how to tell a confirmed relationship from an inferred one, and how to avoid the failure that matters most: a smaller test set that quietly produces false confidence. It also states clearly what TestWarden does and does not do here today, because impact analysis is a topic where claims run ahead of reality.

Test selection does not reduce risk. It moves risk from the tests you skip to your confidence that skipping them was safe.

The real question

Stated carefully, the question is: given this change, which tests could possibly detect a regression it introduces, and how sure am I that I have found them all? The second half is the important one. Any method can list tests that are probably relevant. The danger is the tests it fails to list, because by definition nothing in the run will tell you they were missed.

A selection decision therefore has two outputs: the chosen set and a statement of what the method cannot see. A mature process records both. “We ran the 340 tests linked to the changed requirements and the tests that cover the modified files; we did not account for configuration changes or the shared library” is a defensible record. “We ran the relevant tests” is not.

The main selection methods and what each knows

Most real strategies combine these. Understanding each in isolation makes the combination coherent.

Path and naming heuristics

The simplest method maps changed files to tests by location or name: a change under the payments directory runs the payments tests. It needs no tooling beyond a convention. It is cheap and transparent, and it is wrong whenever a change has effects outside its directory, which is the interesting case. Treat anything from this method as an inferred link.

Static dependency analysis

Build systems and language tools can compute which modules depend on which. A test that depends, directly or transitively, on a changed module is a candidate. Within the boundaries of the build graph this is rigorous. It does not see dynamic dispatch, reflection, runtime configuration, feature flags, database schemas or services reached over the network. Those blind spots are exactly where AI-generated changes sometimes surprise people.

Coverage-based selection

If you record which lines or functions each test executes, a change can be mapped to the tests that touch it. This sees dynamic behaviour that static analysis misses, at the cost of maintaining coverage data, which goes stale as code and tests change and which describes only what was executed under the inputs the tests used.

Requirement-based selection

Jira-centred teams can start from requirements: the change belongs to an issue, the issue has acceptance criteria, and tests are linked to those requirements. This produces confirmed links in the sense that someone recorded the relationship, and it keeps selection tied to what the product is supposed to do. Its weakness is completeness. Requirements describe intended behaviour, not every incidental coupling.

History-based and predictive selection

Past results can guide the choice: tests that have recently failed, tests that failed when similar changes landed, and tests that are slow or flaky. Published industrial work on predictive test selection, such as the Facebook paper on predictive test selection, describes learning from historical change and outcome data to choose tests, and reports substantial savings with a small loss of detection. Two cautions follow. These approaches need a lot of clean historical data, and they are probabilistic: they accept a known miss rate in exchange for speed, which is a deliberate trade that should be a stated policy.

Risk-based and exploratory selection

Human judgement about risk, such as the area is newly written, the author is new, the module has a history of defects, remains an input no tool replaces. It is best used to adjust a rule-based selection, not as the whole of it.

What evidence each method needs

Confirmed versus inferred: an impact map

A responsible tool distinguishes how it knows what it claims to know. The map below is interactive and illustrative: select a changed file to see the candidate regression tests, each labelled as a confirmed link (a recorded relationship such as a requirement link) or an inferred link (a guess from naming or location), along with its recent history. The names, identifiers and counts are invented.

Look at the second file. The ledger test is a confirmed candidate because it is linked to the requirement, and it has failed in two of its last five attempts, so it deserves attention whatever the method. The refund test is an inferred candidate, linked only by shared module, with a clean history. Running it is cheap and sensible. Skipping it would be a judgement that should be recorded as one. The map’s most important property is what it leaves out: it cannot show you the tests that exist for dependencies nobody recorded. That blank space is where false confidence lives.

How smaller test sets create false confidence

False confidence is not a risk of any particular method. It is the predictable psychological result of seeing green on a subset. Five patterns account for most of it.

1. The subset is reported as the suite. A dashboard that shows “all tests passed” for a selected run of eighty out of two thousand gives no hint that most tests did not run.
2. The selector is trusted more as it gets older. Mappings go stale. A coverage map from last year is a map of last year’s code.
3. Selection is evaluated by savings, not by misses. Teams measure minutes saved and rarely measure defects that escaped in unselected tests, because those are hard to attribute.
4. Agents choose their own tests. An AI agent asked to verify its change may run the tests it believes relevant, which are the tests most aligned with its own mental model of the change. Independent selection matters for the same reason independent evidence does; see verifying AI-generated pull requests.
5. No full run ever happens. If subsets are the only runs, accumulated drift goes undetected. A periodic full run is the audit of the selector.

The common remedy is to make the limits of the selection part of the evidence. When a release decision relies on a selected set, the record should say that it was selected, by what method, and when the last full run happened. A decision based on “80 of 2,000 tests, chosen by requirement links and file coverage, with the last full run six days ago” reads very differently from “tests passed”.

Unknown dependencies: the honest category

Every method has a blind spot, and a good practice names it. When a change touches configuration, a shared library, a database migration, an infrastructure definition or a feature flag, none of the file-based methods can be trusted to find the affected tests, and the right response is usually to widen the set rather than to trust the narrow one. Some teams encode this as a rule: changes in certain paths trigger a broader or full run regardless of what the selector proposes. It is a pragmatic acknowledgement that the selector is least reliable exactly where the blast radius is largest.

A worked selection for an agent-authored change

Here is a complete example, invented but typical. An AI agent is asked to make payment retries use exponential backoff, ticket PAY-142. It changes the retry module and, because the schedule needs to be configurable, adds a setting to a shared configuration file. The suite has 2,000 tests and a full run takes ninety minutes. The team’s pipeline must decide which tests to run on the pull request.

1. Stable core. The 120 fast smoke tests always run. They are not selected for this change; they are the baseline.
2. Requirement links. PAY-142 has eleven linked tests, including the retry backoff test. All eleven run. These are confirmed candidates because people recorded the relationship.
3. File-based candidates. Coverage data maps the retry module to nine tests, four of which overlap with the requirement set. Five more run. These are observed links, with the caveat that the coverage data is from the last nightly full run.
4. History. Two tests that touch the ledger posted-entry code failed in the last week. Neither is related by file or requirement, but they run because history is cheap insurance. One of them was flaky and passed on retry; the record says so.
5. Widening on risk. The change adds a shared configuration setting. The rule for configuration changes widens the selection to every test in the payments service, an additional 340 tests. The narrow selector would never have found these, which is the point of the rule.
6. Total. About 480 tests run, roughly a quarter of the suite, in under twenty minutes. The record states the method, the size, the commit and that the last full run was last night.

Now notice what the process did not do. It did not trust the agent’s statement about what it changed. It did not let the selection be only the tests the agent chose to run. The configuration rule widened the net precisely because the change touched something the file-based methods cannot reason about. And it recorded its own limits, so that a reviewer can see this was a partial run.

When the evidence arrives, an independent decision step can compare it with what the change required. If the pipeline timed out and the integration report is incomplete, the change is held for that reason regardless of how clever the selection was. If the report is complete and fresh for the exact commit, and the obligations are satisfied, it clears. Selection narrows what is run; it does not change what must be true of the evidence that comes back. Those are two separate questions, and teams get into trouble when they let the first answer the second.

Reading a selection record

Given a selected run’s record, a reviewer can ask five questions that tell them how much to trust it. Which method chose these tests, and was it appropriate for this kind of change? How large is the set relative to the suite, and has that ratio been stable? When did a full run last happen, and did it agree? Did any widening rule apply, and was it followed? And were failed or flaky attempts kept visible rather than smoothed away? A record that answers all five is worth more than a larger set that answers none, because it lets the reader calibrate.

This is also the right frame for AI-assisted selection when it arrives. A model proposing a set of tests should produce a record with the same properties: what it chose, from which evidence, with its limits stated and its inferred links marked as inferred. If it cannot explain a choice by pointing at recorded history, the choice is a guess, and a guess is best treated as one.

A practical approach in layers

1. Tag tests to requirements. Link each automated and manual test to the Jira issues it verifies. This is the cheapest confirmed-link source and benefits every later method.
2. Run a stable core on every change. A small set of fast, high-signal tests that always runs, regardless of selection.
3. Add selected tests by evidence. Add tests linked to the changed requirements and, where you have the data, tests that cover the modified code.
4. Add history. Include tests that failed recently or that failed on similar changes, and de-prioritize known-flaky tests without hiding them. See flaky tests and release gates.
5. Widen on risk. Configuration, shared libraries, schemas and flag changes widen the set automatically.
6. Run everything on a schedule. A nightly or per-release full run audits the selector and catches drift.
7. Record the selection. Store the method, the set size versus the suite size and the date of the last full run alongside the result.

What TestWarden does and does not do here today

Impact analysis is where marketing most often outruns engineering, so here is the plain position. What exists and has been shown today is the record layer: requirement coverage and release coverage derived from recorded executions and requirement links; version-pinned attempts with full history; failed attempts retained; and Warden decisions that bind evidence to an exact build and hold a change when proof is missing, stale, conflicting or about the wrong build. Those are proven in a Jira Cloud development installation. They give you confirmed relationships and durable history, the raw material for sound selection.

What is not established is automated impact-based selection. Changed-code test-impact analysis, flaky-test and failure-pattern detection and release-risk analysis are implemented in source as advisory analyses over your own history, in a draft integration. No real model usage, cost or acceptance has been recorded for them in the public proof, and they are not counted as an operational capability. A Quality Agent can propose and, where an owner has authorized, take bounded actions, and it never certifies its own work: a repaired change needs fresh independent evidence. TestWarden does not today choose which tests your pipeline runs, and it does not claim a miss rate, a time saving or any benchmark, because none has been measured on customer data. See Quality Memory for the underlying history.

Measuring whether your selection is safe

• Escape rate by selection state. For defects found later, was the test that would have caught it selected? This is the number that matters and the hardest to collect.
• Full-run disagreement. When the periodic full run fails and the selected runs passed, record which tests failed and why they were not selected. Each is a selector bug.
• Selection ratio over time. If the selected fraction keeps shrinking without a matching rise in full-run confidence, you are drifting.
• Unmapped tests. The count of tests with no requirement link. They are invisible to requirement-based selection.
• Stale mappings. How old your coverage and dependency data is.

Frequently asked questions

Is it safe to run only the tests related to the change?

It is safe to the extent that your relationships are complete and current. Treat the result as evidence about the selected set, record the method, and audit it with periodic full runs.

Does AI-generated code need more regression testing?

It needs independent evidence at least as strong, and often broader selection, because the author’s understanding of side effects may be thin and the same agent may have written the tests. Volume also grows, which is why principled selection matters.

Can AI choose the tests for me?

It can suggest candidates. An agent choosing the tests that verify its own change is a conflict of interest, so the selection that counts as evidence should come from a source the author does not control.

How do I handle flaky tests in selection?

Track them, deprioritize without deleting, and keep their attempts in history. Never let a flaky test’s retry hide a real failure.

Does TestWarden select tests?

Not today. It records coverage, history and decisions. Automated selection analyses exist only as advisory source under acceptance and are not counted as capabilities.

What is the minimum I should record about a selected run?

The method, how many tests ran versus the suite size, the commit, and the date of the last full run. With those four, a reader can judge how much the result means.

What is the difference between inferred and confirmed links?

A confirmed link is a recorded relationship, such as a requirement linked to a test in Jira. An inferred link is a guess from naming, location or similarity. Both are useful, and they deserve different weight.