How to Independently Verify AI-Generated Pull Requests Before Merge

If you use Cursor, Claude Code, GitHub Copilot or Codex to write pull requests, you have already met the awkward moment. The agent finishes, the checks are green, and a reviewer is asked to approve a diff they did not write and a test suite the same agent just modified. The honest question is not whether the code looks plausible. It is what independent evidence exists that the change does what it should.

This article is a practical answer for teams that already ship AI-assisted changes. It does not argue against using agents. It describes an architecture in which an agent can be as productive as you like while the question “is this enough to merge?” is answered by something the agent does not control. It also marks clearly where code review stops being able to help. For the shorter conceptual argument, see why coding agents should not certify their own changes; this piece is the how-to.

The problem is agreement, not intelligence

It is tempting to frame AI code verification as a question of model quality: better models write fewer bugs. That is partly true and beside the point. Even a perfect author needs an examiner, because the author’s misunderstanding of the requirement is invisible to the author. A human developer who misreads a ticket writes code and tests that both reflect the misreading. An agent does the same, faster, and in volume.

What changes with agents is the economics of review. A human can read a ten-line change carefully. An agent can produce a hundred changes a day, each with a confident summary. Reviewers adapt by skimming the summary, trusting the green check and sampling. Every one of those adaptations moves trust from evidence to the author’s own report. The architecture below moves it back.

The agent’s statement that all tests pass is a claim. Evidence is what an independent system recorded about the exact commit.

What actually goes wrong: adversarial scenarios

You do not need to assume malice. These are ordinary failure modes that arise when a system optimizes for “checks go green”. Treat each as a scenario your verification must survive.

Notice the pattern in the last column. Every item there is a signal the author can influence. The middle column is made of things the author cannot influence, or can influence only visibly. That is the design principle: evidence that counts must come from a producer the change cannot reconfigure, and be tied to the exact thing being merged.

The path from agent to decision

The step-through below shows the architecture end to end and lets you try the one decision that matters: what happens when the only thing received is the agent’s own success claim, versus independent evidence for the exact build.

Code review versus test evidence

People often argue as if review and testing were substitutes. They answer different questions, and AI-generated changes make the difference sharper.

Use both, and be explicit about which question each answers. One practical consequence: do not let an AI reviewer be the only approval. A reviewer model can share the author’s blind spots and accept behaviour that is plausible but wrong, so it is a useful second reader and a poor sole gate. Published guidance for reviewing AI-generated code says much the same: review the diff against the requirement first, inspect new dependencies, configuration and generated tests with extra attention, and run the focused tests and then the full required checks.

A concrete verification architecture

Here is an architecture you can build with ordinary tools. It has four layers, each defending against a different scenario from the table.

Layer 1: protect the things that define “passing”

Tests, snapshots and pipeline configuration are not ordinary code in an AI workflow. They are the definition of success. Give them stronger protection than the code they check. In GitHub that means branch protection with required status checks, and code owners so that changes to test directories and workflow files require review by someone other than the author. A minimal CODEOWNERS file:

# Changes to what defines "passing" need an owner who is not the author
/tests/              @acme/qa-owners
/.github/workflows/  @acme/platform
**/__snapshots__/    @acme/qa-owners

This does not stop an agent from proposing a change to a test. It makes the change visible and puts a human who owns the suite in the loop, which defeats the quietest failure modes: weakened assertions, deleted tests and edited pipelines.

Layer 2: produce evidence in CI the agent does not control

Run the full suite in a pipeline whose configuration is protected, with a token the agent never holds, and have it write a complete report in a standard format. JUnit XML for tests and SARIF for scanners are the common choices. A minimal job that produces and keeps a JUnit report, using Playwright as an example runner:

name: verify
on:
  pull_request:
permissions:
  contents: read
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test --reporter=junit
        env:
          PLAYWRIGHT_JUNIT_OUTPUT_NAME: test-results/junit.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: junit
          path: test-results/junit.xml

Two details matter. if: always() keeps the report when tests fail, because a failure you cannot see is the worst outcome. And the report is an artifact tied to a run, which is tied to a commit. That tie is what the next layer uses. The GitHub Actions deep dive covers build identity and report completeness in detail.

Layer 3: decide from evidence, not from the checkmark

A green check on a pull request says that some jobs exited zero. It does not say that the right jobs ran, that they were complete, or that they describe the commit you are about to merge. A separate decision step reads the evidence and applies a written policy: which obligations must be satisfied, how fresh the evidence must be, and what to do when it conflicts. This is the role of The Warden: it receives the reports with the repository, commit, build and environment they describe, evaluates the verification contract, and records HELD or CLEARED for that exact change. Evidence about a different commit does not count, a failing or stale or mixed set holds the change, and a payload that tries to assert its own success is rejected.

Layer 4: keep the history

When the agent pushes a fix, the new commit needs new evidence and a new decision. The earlier HELD decision should remain, because the record of what failed first is exactly what a later reviewer, or the next agent, needs. Append-only history also stops the quiet rewriting that turns a failing record into a passing one.

A policy for agent-authored pull requests

Write the policy down before the next agent run. The one below is a starting point you can adapt; it is a process recommendation, not a TestWarden feature list.

1. Identity: every report names the repository, commit, build and environment. Reports that do not are ignored.
2. Independence: at least one test report and one scan report come from a pipeline whose configuration the change cannot modify without an owner’s approval.
3. Completeness: the report is complete for the expected suite. A drop in executed tests versus the base is unresolved until explained.
4. Freshness: evidence must be for the current head commit. A newer push supersedes older evidence.
5. No self-certification: the agent may propose, summarize and dispatch authorized runs. Its statements are never evidence, and it holds no token that can write a decision.
6. Human authority: exceptions are decided by a named person and recorded with a reason.

What the agent can and should do

None of this makes agents less useful. In a governed workflow they can read the reasons a change was held and repair it, propose additional tests for a human to accept, explain failures in plain language, and, where an owner has authorized it, trigger a verification run. What they cannot do is decide. In TestWarden’s own development acceptance (dated 2026-10-27, installed development version 3.18.0), a Quality Agent dispatched a real workflow; a separate protected worker then fetched the actual run and native Check independently, and Forge read both the new and the original decisions. The original HELD decision stayed on record. The agent never cleared anything; the policy did, from independent readback. That acceptance is an engineering result in a development installation, not a customer outcome, and the same record is clear that the overall release is not yet customer-ready.

A reviewer’s protocol for an agent-authored pull request

Architecture handles the mechanical risks. The human reviewer still carries the judgement calls, and a consistent order of reading helps more than extra effort. This is the sequence that tends to work, ordered so that the most agent-specific risks are looked at while attention is fresh.

1. Read the requirement first, not the summary. Open the Jira issue and its acceptance criteria before the pull request description. The agent’s summary is a hypothesis about what the change does; the issue is what it was supposed to do.
2. Look at the test diff before the code diff. Ask whether any assertion got looser, any test got removed or skipped, any expected value was edited, or any mock now stands where real behaviour used to run. A change that touches tests and production code together deserves more suspicion than one that adds tests only.
3. Check what is new at the edges. New dependencies, configuration, migrations, permissions and network calls are where agents are most likely to add something nobody asked for.
4. Compare scope with intent. Agents often make helpful adjacent changes. Each is a risk that was never planned or tested. Ask for them to be split out.
5. Open the evidence, not the badge. Confirm which commit the report names and whether the run completed. If your tooling shows a decision with reasons, read the reasons.
6. Ask the one question that matters. If the author’s summary were wrong, which piece of independent evidence would reveal it? If you cannot name one, the change is not verified; it is merely unchallenged.

This protocol is not specific to any tool. Its purpose is to point human attention at the places where AI-generated changes differ from human ones: confident summaries, edited tests and unrequested scope. It also shows why independent evidence is not a replacement for review. A reviewer who spots a loosened assertion has found something no report would have flagged.

A worked example: the retry that passed for the wrong reason

The following example is invented, but each step is a pattern teams report with agent-authored changes. A developer asks an agent to make payment retries use exponential backoff. The agent edits the retry module and runs the tests. One existing test fails: it asserts that the second retry happens after one second, and the new code waits two. The agent updates the expected value to two seconds, reruns, and reports success with a tidy summary.

In a pipeline that only reports “checks passed”, everything is green. A reviewer skimming the summary sees an expected change in behaviour and an updated test, and approves. What happened is that the test, which encoded the product’s actual requirement of retrying quickly enough to avoid a customer-visible timeout, was changed to match the code instead of the other way around. Nothing in the green check can reveal that.

Now run the same change through the architecture above. The test directory is covered by code owners, so the edit to the expected value requests a review from the test owner, who recognizes it as a behaviour change and asks the author to confirm it against the requirement. The requirement says retries must complete within a bounded window; the new schedule does not. The change is sent back. Separately, the protected CI pipeline produced a complete report for the head commit and the pipeline evidence is complete, but the merge waits for the owner’s review of the test change.

The agent then proposes a schedule that satisfies the requirement and restores the original test. New commit, new complete evidence, new decision. The earlier decision, recorded against the first commit, remains in the history together with the reason it was held. Two weeks later, when someone asks why the backoff schedule looks the way it does, the pull request thread, the held decision and the repair tell the story without anyone needing to remember it.

The lesson is not that the agent was careless. It did exactly what its objective, make the tests pass, asked for. The lesson is that an objective of “make checks green” is satisfied equally well by fixing the code or by changing the check, and only an independent definition of passing can tell the difference.

What to measure once this is running

Measuring is optional, but a few signals tell you whether the architecture is working and whether trust in agent output is rising or falling for good reasons. Track them as trends on your own data; this article does not claim benchmark values and none are available.

• Held-then-repaired rate. How often a first submission is held and later cleared. A very low rate on agent-authored changes may mean the bar is too loose.
• Test-diff frequency. How often agent pull requests modify existing tests, and how often those modifications are approved. A sudden rise is worth a look.
• Time to repair. How long a held change takes to be repaired and re-verified. If it is long, the reasons given are probably not actionable enough.
• Evidence incompleteness. How often runs produce partial reports. This is usually an infrastructure problem hiding as a quality problem.
• Exceptions granted. How many conditions were waived by a named person, and how many were later regretted.

Limits and honest caveats

• Independent does not mean correct. If the requirement is wrong or the tests are shallow, independent evidence will faithfully certify the wrong thing. Test design still matters.
• Coverage gaps remain gaps. Evidence shows what ran. It is silent about what nobody wrote a test for.
• Default contract only. Today customers use TestWarden’s default verification contract. Customer-authored contracts, waivers and approval options are not built.
• Marketplace availability. TestWarden is publicly listed on Atlassian Marketplace. The verification examples in this article were demonstrated in development rather than a verified customer production installation. Everything described as working was observed in development installations.
• Not a scanner or pipeline. TestWarden judges evidence; it does not replace your test runner, scanner or CI.

A staged rollout for a team already using agents

1. Week 1: protect the definition of passing. Add code owners for tests and workflows. This costs almost nothing and stops the quietest failures.
2. Week 2: make evidence complete. Ensure CI always writes and keeps complete reports, including on failure.
3. Week 3: bind to identity. Record the commit and run next to every result, and refuse to cite results from other commits.
4. Week 4: add the independent decision. Introduce a policy step, by hand or with a tool, that reads the evidence and records a decision. Start in report-only mode, comparing its decisions with what reviewers would have done.
5. Then tighten. Move from advisory to required once the false-hold rate is understood.

Frequently asked questions

Is it enough to have a human review every AI pull request?

Review is necessary and not sufficient. It is excellent at design and hazard spotting and weak at exhaustive behaviour, and it degrades with volume. Pair it with independent test evidence, and be sure tests and pipeline configuration are protected.

Can another AI model review the first model’s code?

It can be a useful extra reader. It should not be the only approval, because models can share blind spots and accept plausible but incorrect behaviour. Keep a deterministic evidence-based gate.

What if the agent needs to change a test legitimately?

Let it propose the change. Route it to the test owner for approval, and require that the evidence for the new commit is produced by the protected pipeline.

Does TestWarden run my tests?

No. Your CI runs them. TestWarden receives the resulting JUnit, SARIF or JSON evidence with the build identity and decides whether it satisfies the verification contract.

Can an agent clear a change in TestWarden?

No. Evidence payloads that carry a caller-chosen state, progression flag or approval are rejected, and agent-supplied evidence cannot satisfy the requirement for a CI run. The decision is computed by policy.

Is this live for customers?

TestWarden is publicly listed on Atlassian Marketplace. The verification evidence described here was proven in development installations with real CI runs. See Proof for exactly what has and has not been shown.

We use Xray or AIO Tests. Does this replace them?

No. They remain your test management. The decision layer sits above them and other evidence producers; see test management versus release governance.