Building an Independent CI Quality Gate with GitHub Actions, JUnit, SARIF and Jira

Most pipelines already have something they call a quality gate: a required status check that turns red when a job fails. That is a good start and a weaker guarantee than it looks. A job turning green tells you that a process exited with status zero. It does not tell you that the tests that matter ran, that they ran to completion, or that they ran against the commit you are about to ship.

This article is for platform and DevOps engineers who want a gate that answers those questions. It covers what to bind evidence to, how to produce reports that are complete enough to trust, how to avoid a subtle trap with pull-request commits, and how an independent decision step reads the result and reports back to GitHub and Jira. The examples use GitHub Actions because it is common, but the principles carry to other CI systems. Where we describe TestWarden’s behaviour, the source is the public CI evidence documentation, and its limits are stated in plain terms.

What “independent” means in a pipeline

A gate is independent when the party being judged cannot change the judgement. In CI that has three concrete meanings. The pipeline definition is protected, so the change under test cannot edit it without a separate owner’s review. The credentials that submit evidence are held by CI and never by a coding agent or a developer’s laptop. And the decision is computed from submitted evidence by a policy, rather than by a job that the author’s code can influence.

If any of those fail, the gate degrades into a self-report. A pull request that modifies its own workflow file to skip a failing job will produce a green check that is perfectly truthful about the pipeline that ran, and says nothing about the code. The remedy is organizational as much as technical: protect workflow files with code owners, require status checks through branch protection, and treat any token that can write evidence like a deployment credential.

The shape of the gate

The diagram below traces the flow for a failing first attempt and the repaired build. Switch between the two scenarios to see what changes: the commit identity, the completeness of the integration report, the decision and the status reported back. The values are invented.

Read it left to right. Build identity names the exact change. Producers write evidence in standard formats. Obligations are the things the verification contract requires. The decision is computed from the first three. Finally the outcome is reported to GitHub as a Check and to Jira, and a gate script converts the decision into an exit code so the pipeline itself can stop. The rest of the article takes each stage in turn.

Binding evidence to an exact build

Every report must name what it is about: the repository, the commit, the build and the environment. GitHub Actions gives you most of the raw material in default environment variables and the workflow context. These are the ones that matter for a gate:

The pull-request merge-commit trap

On pull_request events, the default commit that GitHub checks out and exposes as GITHUB_SHA is generally a temporary merge commit that combines the pull request with its target branch, not the commit at the tip of the author’s branch. The head commit is available in the event payload as github.event.pull_request.head.sha. Which one you bind evidence to is a decision with consequences. Binding to the merge commit describes what would result from merging now, and changes when the base branch moves. Binding to the head commit describes the author’s work exactly, and does not change when the base moves.

Neither is wrong, but mixing them is. A common bug is a pipeline that builds the merge commit but reports tests against the head commit, so an evidence record names a commit that was never actually built. Decide deliberately, document the choice, and make sure the commit named in the evidence is the commit that was checked out. GitHub’s own documentation on workflows describes the events and context involved; read the section for the event types you use.

Job outcomes are not evidence

A common pattern is to gate on the job conclusion: if the test job succeeds, the check is green. That treats the exit code as the evidence. It is fragile in ways that matter once the pipeline gets any complexity.

• Exit zero with nothing run. A misconfigured glob that matches no tests can exit successfully. The pipeline is green and nothing was tested.
• `continue-on-error`. A step marked this way can fail while the job reports success. Do not use it on a step that produces evidence for a gate.
• Retries that hide failures. Runner-level retries are fine for infrastructure noise, but a test that fails then passes on retry should leave a record. Otherwise the report shows only the pass.
• Sharded runs. A matrix of shards produces one report per shard. If only the last uploaded report is submitted, the evidence describes a fraction of the suite.
• Timeouts. A job killed at its time limit may leave a partial report that lists only what finished.

The fix is to treat the report as the evidence and the job outcome as a hint. Always upload the report, including on failure, with if: always(). Merge shard reports into one before submitting. Compare the number of executed tests against an expectation so a partial run is noticed. Documentation for TestWarden’s gate says this directly for multi-spec runs: a multi-spec Cypress run writes several files and they must be merged into one complete JUnit report, because submitting only the last spec and calling it the complete run would hide failures.

Complete reports: JUnit and SARIF

JUnit XML for tests

JUnit XML is the lingua franca for test results. Most runners, including Playwright, Cypress, pytest and Maven’s Surefire, can write it. For gating, completeness is the property to protect: one report that represents the whole run for the build. TestWarden’s evidence endpoint rejects reports that carry document-type or entity declarations, and limits each request to 256 KiB, with a lossless compressed path for larger JUnit reports up to 2 MiB of raw XML. A report beyond that stays held with an explicit reason rather than being truncated.

SARIF for scanners

SARIF, the Static Analysis Results Interchange Format, standardizes scanner output; the OASIS SARIF 2.1.0 specification is public, and GitHub documents how code scanning consumes it in its SARIF support guide. Two properties matter for a gate. The run must record that the analysis completed successfully, because an interrupted scan can emit an empty list of findings that looks clean. And each finding must be treated according to its kind: informational and not-applicable results are not failures, but an ordinary finding should hold the scan until it is resolved or accepted. TestWarden requires SARIF 2.1.0 with an explicit successful invocation and holds the scan on any finding other than pass, informational or not applicable.

Which evidence satisfies which obligation

It is worth being explicit that different evidence answers different obligations. A clean scan does not show that tests ran, and passing tests do not show that a scan was clean. TestWarden’s documentation spells out the default relationships:

The gate step and what it does with the decision

With reports in hand, the gate step submits them, waits for the decision and converts it into an exit code. TestWarden’s documented gate is a script, scripts/warden-gate.mjs, supplied during onboarding, that exits zero only when the response is a structurally valid decision of CLEARED that matches the exact repository, commit, build and environment submitted. Every other outcome, including held, verifying, incomplete, needs approval, unknown, malformed or unauthenticated, exits one. Invoked directly, it looks like this, taken from the documentation:

node scripts/warden-gate.mjs --file test-results/playwright.xml --format junit --json
echo "exit code: $?"

Inside GitHub Actions, the repository, branch, commit, pull request, run, attempt and build are derived automatically; outside it you set the identity values yourself. The reusable composite action warden-verify wraps the same gate, announces the run, writes a job summary listing the obligations to resolve, and keeps a single pull-request comment current. Its documented step looks like this:

- name: Submit evidence to The Warden
  uses: ./.github/actions/warden-verify
  with:
    endpoint: ${{ vars.WARDEN_ENDPOINT }}
    environment: development
    evidence-file: test-results/junit.xml
    format: junit
    access-token: ${{ secrets.WARDEN_TOKEN }}

Reruns: run the whole job

A subtle point for anyone used to “re-run failed jobs”. Re-running only the gate for an earlier attempt resubmits that attempt unchanged, so it cannot produce a different result. Evidence for an older build than the current target returns a superseded error and can never clear the change. After a fix, push the new commit and let the entire pipeline run again, so the evidence for the new build is genuinely new. Exact retries of the same submission are idempotent and keep the original observation time, while changed evidence is stored as a new receipt and conflicting evidence is never overwritten.

Reporting back: GitHub Checks, Jira and independent readback

A decision that nobody sees has not governed anything. Two surfaces carry it. In GitHub, a Check gives the pull request a visible status you can require in branch protection; GitHub’s Checks API is the mechanism. The composite action can create a check run when enabled, which needs the checks: write permission. In Jira, the decision and its reasons are recorded against the change so the people who own the release can read them where they already work.

The word independent applies to the readback too. In TestWarden’s own development acceptance, dated 2026-10-27, a native Check for an exact source, run and attempt was fetched with a separate GET and confirmed as success from the GitHub Actions app, separately from the decision that produced it. That kind of readback is what lets you trust the status you see: it is the platform’s record, not an echo of what a script claimed. Treat a status you cannot read back independently as a claim.

Failure modes and what each looks like

Permissions, secrets and fork pull requests

A gate that submits evidence needs a credential, and credentials in pull-request pipelines deserve more care than most teams give them. Start from least privilege: declare permissions at the workflow level and grant only what each job needs, with contents: read as the default and checks: write or issues: write added only to the job that creates a check or a comment. Keep the evidence token in the platform’s secret store and never in command arguments, source control or logs; the TestWarden gate redacts it from its own output, but your other steps will not.

Fork pull requests are the awkward case. By design, workflows triggered from forks do not receive repository secrets, so a gate that needs a token cannot submit evidence for an untrusted fork. Resist the temptation to work around this by running untrusted code in a privileged context; GitHub’s security guidance on that pattern is worth reading before you try. The safer arrangement is to run the untrusted tests without secrets, store the reports as artifacts, and have a separate trusted workflow, with the token, read those artifacts and submit them. Whatever you choose, remember that the gate’s value comes from the evidence being collected by a pipeline the change cannot reconfigure, so the more privileged part should never execute the contributor’s code.

Designing the obligations list for a service

The gate is only as meaningful as the list of things it requires. Teams new to this tend to start with one obligation, “tests pass”, and then discover the list should be a little richer. A reasonable starting set for a typical service is short enough to keep and specific enough to matter:

• A complete automated test report for the build, from the protected pipeline.
• A completed scan report for the same build, with no unresolved findings above your threshold.
• Evidence freshness. Anything older than the window you choose is stale. TestWarden’s default contract treats evidence older than 24 hours as stale; if your cadence is different, that is a conversation to have with the contract in mind rather than a number to copy.
• An environment match. Staging evidence does not authorize production.

Write the list in plain language first, and only then in configuration. It is the artifact a reviewer, an auditor and a new engineer will read. Today TestWarden customers use the default contract, so for now this exercise informs what you submit and how you interpret reasons, not a contract you author yourself.

Honest status: what is real today

It would be easy to oversell this as an effortless, drop-in step. It is not, yet, and the accurate picture is useful for planning.

• Marketplace availability. TestWarden is publicly listed on Atlassian Marketplace. This article cites development validation rather than independently verified customer production deployments.
• Setup involves your Jira admin. A site administrator enables App REST APIs and access for your CI is provisioned, which we can assist with during guided onboarding. The gate script and the warden-verify action are delivered during pilot onboarding, and the action is a developer preview.
• What was proven. Real GitHub Actions runs against TestWarden’s own repository submitted JUnit evidence to a development installation: a failing build was HELD, and the fixed build was CLEARED with the HELD decision kept. SARIF and JSON are covered by automated tests; a live submission from a real scanner has not been run.
• Test-management collectors. Read-only collectors for AIO Tests, Xray, Zephyr Scale, TestRail and Azure DevOps exist in source and are not vendor-certified. Today, what reaches the gate from those systems is a complete JUnit report your CI produces.
• Default contract. Customer-authored verification contracts are not available.

A gate-readiness checklist for your pipeline

1. Workflow files and test directories are protected by code owners and required reviews.
2. The evidence-writing token is a CI secret that no agent or developer workstation holds.
3. Every report is uploaded with if: always(), so failures leave evidence.
4. Sharded or multi-file reports are merged into one complete report before submission.
5. The commit named in the evidence is the commit that was checked out and built, and the merge-commit versus head-commit choice is documented.
6. The run and attempt identifiers travel with the evidence.
7. The gate step fails the job on any non-CLEARED outcome and is not set to continue on error.
8. Re-running after a fix runs the whole job on a new commit.
9. The status is required in branch protection, and you can read it back from the platform.
10. A scan report exists for the same build and records a successful run.

Frequently asked questions

Isn’t a required status check already a quality gate?

It is a gate on a job outcome. That is useful, and it is silent about report completeness, build identity and whether the right jobs ran. An independent decision step reads the evidence and applies a policy.

Which commit should evidence name on pull requests?

Whichever commit your pipeline actually builds, stated consistently. By default GitHub checks out a temporary merge commit for pull_request events; the author’s head commit is in the event payload. Decide, document, and make the evidence match.

Do I need TestWarden to build a quality gate in Actions?

No. You can implement identity binding, complete reports and a required check yourself. TestWarden adds a policy decision, append-only history, Jira context and the fail-closed handling of stale, conflicting or wrong-build evidence.

Does it work with Playwright, Cypress or Semgrep?

Those are not first-party integrations. They work because they can write JUnit XML or SARIF. Any tool that emits those formats is compatible, and a tool’s own green result is evidence, never a clearance.

What about GitLab, Jenkins or other CI systems?

The principles are the same. Outside GitHub Actions you set the repository, commit, build and run identifiers yourself. Only GitHub Actions has been exercised with real runs so far.

Can the agent that wrote the change submit the evidence?

It should not hold the token. In TestWarden, only CI holds the token, and agent-supplied evidence cannot satisfy the CI-run requirement.

Is the integration with Xray or AIO Tests live?

No. Collectors are implemented in source and awaiting live vendor acceptance. See Integrations for the status of each provider.