TestWarden documentation

Start here when you have a job to do. Developers can also begin at the developer overview.

• Onboarding

  Choose a path, enable access, confirm the first change appears.

• CI evidence

  Submit JUnit, SARIF or JSON and gate on the exact build.

• API, CLI and MCP

  Read status, obligations and explanations from a terminal or agent.

• Decisions and approvals

  States, reason codes, next actions and preserved history.

• Permissions

  Roles, custom scopes and who can read, write or request.

• AI and data handling

  What is optional, what is consented and what never changes a decision.

Availability labels: Marketplace V1 — publicly listed covers native test management, migration, evidence intake, the CI gate and Warden decisions, submitted to the Atlassian Marketplace. Developer preview — live acceptance pending covers the Developer API, PowerShell client, MCP server, GitHub action and optional AI explanation. These are a developer preview: access is arranged during pilot onboarding and live acceptance is still pending.

TestWarden is a Jira Cloud app built on Atlassian Forge. Its core is a verification control plane that sits above the tools your team already uses; it also includes native Jira test management. You can use either part, or both.

• Verification control plane (The Warden): CI systems and other verifiers submit evidence for a specific change as JUnit XML, SARIF 2.1.0 or JSON. The Warden evaluates that evidence against a verification contract under a deterministic policy and records a Quality Decision: CLEARED VERIFYING HELD NEEDS APPROVAL or INCOMPLETE. Keep your existing tools and add The Warden above them.
• Native test management: a shared test repository, structured steps, test cycles, step-by-step execution with attempt history, Jira defect creation, requirement coverage and a Migration Center, for teams that want native test management, either on its own or next to the tools they already run.

See The Warden for how decisions are made and Security for scopes and architecture. TestWarden is the brand. The Warden is the verification authority you work with inside the product.

• A Jira Cloud site. A Jira administrator installs TestWarden from the published Atlassian Marketplace listing and accepts the requested app scopes.
• TestWarden requests two Jira scopes: read:jira-work (project and issue discovery, requirement links, coverage, permission lookup) and write:jira-work (user-initiated Jira Bug creation from a failed or blocked step). It also declares report:personal-data, which Atlassian requires for personal-data reporting. See Security.
• For CI evidence submission, a site administrator enables App REST APIs for TestWarden in Atlassian Administration → Connected apps. Access uses OAuth 2.0 (3LO) with the app-defined scopes (see Permissions) write:warden-evidence:custom (submit evidence) and read:warden-changes:custom (read a change and its latest decision). For guided CI setup and pilot onboarding we can help provision access; see Onboarding.

Roles

TestWarden maps your Jira permissions to app roles. Viewer reads. Tester executes tests and links defects. QA Manager authors tests, saved views, folders, cycles and links. QA Admin can additionally archive, restore and run migrations. Operations that are not classified fail closed to Admin.

1. As a Jira site administrator, install TestWarden from the Atlassian Marketplace listing and accept the requested scopes.
2. Open Apps → TestWarden. The app opens in the Changes workspace. Until CI is connected it shows: “No changes received yet. Connect your CI through the App REST API to begin collecting verification evidence.”
3. On an empty repository the Welcome to TestWarden banner asks “Where are your tests today?” Choose We already track tests elsewhere to go to the Migration Center, or We're starting fresh to create your first test.

The top navigation is Changes · Repository · Plans & Runs · Verification · Migration · Evidence Sources. The ? button opens Help and feedback, including a diagnostic reference you can copy into a support request.

The Repository has two scopes. Shared Library holds canonical tests that can be associated with many Jira projects without copying them. Project Tests are tests that keep a required Jira project association. Archived shows recoverable archived tests.

• Search and quick views: search by title; toggle Mine, Unassigned, Ready and Recently updated.
• Filters: priority (Low, Medium, High, Critical), status (Draft, Ready), owner, tag and Jira project. Active filters appear as chips; Reset clears them.
• Saved filters and folders: save the current filter set from the sidebar; create nested folders to organise without duplicating tests.
• Bulk actions: select rows, then use Actions ▾ to set status or priority, move to a folder, assign to me, clear owner, add or remove a tag, or Archive. In Archived view the action is Restore selected.
• Rows show the test key (for example TW-3F9A12BC), version, tags and associated projects. Large libraries load with indexed cursor pagination and Load more.

1. Click + New test, enter a title (and optional description / preconditions) and choose Create & edit. The editor opens immediately.
2. Set Priority, Status (Draft or Ready), Folder, Owner and Tags. Shared tests can link additional Jira projects; link Jira issues under Requirements to drive coverage.
3. Add steps with Action, Test data and Expected result. Use Ctrl/⌘ + Enter to add a step. Each step supports Edit, Insert below, Duplicate, move up/down and Delete.
4. Paste steps accepts one step per line with tabs separating Action → Test data → Expected result, up to 100 steps at a time. A test holds up to 500 steps.

Edits autosave about 1.2 seconds after you stop typing; the header shows Saved, Saving… or Unsaved changes. Undo and Redo cover draft field edits. Every saved change creates an immutable version visible under Version history. Use Clone test to copy a test, and Archive (recoverable) rather than deletion.

Create a cycle

In Plans & Runs, create a cycle with a name (for example “Release 4.3 regression”), optional release/version, environment and Jira project. A cycle can hold up to 500 canonical tests without copying them.

Add tests

Select tests in the Repository and choose Add selected to cycle, or use Compose from… a Folder, Saved view or Tag. Tests already in the cycle are skipped.

Run a test

• Choose Run on a queued test. The runner pins the current test version (shown as “Attempt n · Test vN”) and snapshots it, so later edits never change this execution.
• Mark each step Passed, Failed, Blocked or Skipped (shortcuts P F B S; N resets to Not Run). Pass all remaining marks undecided steps Passed.
• Finish the execution as Passed, Failed, Blocked or Skipped. Passed is unavailable while any step is Failed, Blocked or Not Run.
• Run again starts a new attempt. History shows every attempt with its status, test version and time. Executed tests stay in the cycle to preserve coverage history.

The cycle header shows percent executed and counts for Passed, Failed, Blocked, In Progress, Not Run and Skipped. Export CSV downloads execution results with provenance; enable All attempts to include every attempt.

When a step is marked Failed or Blocked, record the Actual result and an optional comment, then choose Create Jira Bug. Pick the defect project and issue type (Bug by default); the summary defaults to “[TestWarden] <test title> — step n Failed”. Cycle, release, environment and attempt are auto-filled into the Jira description together with the step’s action, test data, expected and actual results.

The created issue appears under Linked Jira defects on the step and persists with the execution. If the issue is created but linking fails, use Retry link.

Open Verification. Coverage is derived from recorded executions and Jira requirement links; no JQL or dashboard setup is needed.

• Requirement lookup: choose a Jira issue (type at least two characters; Jira returns only issues you can view) to see linked tests, their latest attempt and status, and a coverage percentage.
• Release coverage: scope by cycle, project key or release ID to see execution completeness, failing and blocked tests, a per-cycle progress breakdown and which candidate requirements have no linked test.

Keep AIO Tests, Xray, Zephyr Scale, TestRail, Azure DevOps and your CI. The Warden treats their results as evidence for an exact change; it does not replace them and it does not move any test case.

• Integration. keep your original test-management platform and use its results as verification evidence for The Warden. No test case is moved.
• Migration. optionally import supported test cases into TestWarden’s native Jira repository. Execution history stays in the original tool.

Read-only integrations for AIO Tests, Xray, Zephyr Scale, TestRail and Azure DevOps are being completed and verified for TestWarden’s independent governance workflow. Early-access integration testing is available by arrangement, subject to readiness.

CSV migration brings your test cases, not historical executions. Open Migration (Migration Center) and choose a source: Move from Xray (Xray Test Case Importer CSV or a Jira/Xray CSV with Manual Test Steps JSON), Move from Zephyr Scale (Zephyr Scale CSV) or Generic CSV. All three are labelled Ready. Ready means verified on a development Jira site against documented export layouts, not against every Xray or Zephyr edition or export, so always review the preview before committing. AIO Tests has no dedicated importer: export your cases to CSV and use Generic CSV with manual column mapping. Migration requires the QA Admin role. See also Migration.

1. Upload a CSV file (up to 10 MB; up to 10,000 rows per job).
2. Map fields. Vendor exports are detected automatically (“Zephyr mapping detected”) and the preservation rules are shown. For Generic CSV, map each column to Title (required), Description, Priority, Status, Tags, Steps, a new or existing custom field, or Ignore.
3. Preview & validate. Review Will import, Will skip and Total rows, the skipped-row reasons and a sample of mapped rows. Re-previewing the same file and mapping reuses the existing job.
4. Import. Choose Commit import. The import runs in the background in batches with checkpoints; you can close the panel and it resumes where it left off. Use Cancel import to stop.
5. Verify & report. Filter rows by Imported, Imported (with warnings), Skipped or Failed, read the reason for each, and open imported tests directly. Imported tests carry the label “Imported from migration”.

The Changes workspace lists every Change Set TestWarden has received, identified by repository and commit, with the current Warden state, build and environment. Open a change to see:

• the decision with its contract version, policy version and engine version, grouped Failures, Missing proof, Conflicts, Stale evidence and Required approvals, plus Reasons and Next actions. The reason codes are evidence_satisfied, evidence_failed, only_stale_evidence, no_matching_evidence, conflicting_evidence, open_blocking_defect, missing_required_approval, verification_run_in_progress, verification_run_overdue, no_conclusive_evidence, no_active_contract, authoritative_defect_context_unavailable and evidence_set_truncated;
• Evidence received: each receipt’s producer, result, type, version, run, build, environment and message, with provenance (receipt id, content digest and raw report reference);
• Decision history: earlier decisions are preserved and never rewritten.

The default contract requires a complete, current-build automated test run no older than 24 hours with no failed, error or skipped observations. Scanner or reviewer passes are recorded as evidence but cannot satisfy the test obligation on their own. Customer-authored contracts, approval and waiver options and defect policy are not customer-configurable in the launch version. Recorded manual executions are not yet inputs to Warden decisions. More in Decisions and approvals.

Evidence is submitted over the TestWarden Forge App REST API (POST /warden/evidence) using OAuth 2.0 (3LO) with the app-defined scopes. Supported formats:

• JUnit XML — complete test-run reports from CI (format: junit, provider type ci).
• SARIF 2.1.0 — static-analysis and security results (format: sarif, provider type scanner).
• Generic JSON — an envelope with provider, change (repository, commit, build, environment), correlationId, observedAt and 1–20 observations with results passed, failed, error or skipped. Provider types: ci, agent, scanner, code_reviewer, visual_agent.

Each submission is limited to 256 KiB and must name the exact repository, commit, build and environment it verifies. Exact retries are idempotent; conflicting evidence is never overwritten. Only CI credentials may announce a verification run. Submissions that include a caller-selected state, progression flag or approval are rejected. The Evidence Sources workspace lists accepted formats and which producers have submitted accepted evidence. Results from Xray, AIO Tests, Zephyr Scale, TestRail or Azure DevOps are not synchronised. Until the read-only connectors described under Integrations are accepted, they reach The Warden only if your CI emits them as one of the formats above.

• Free: Jira sites with up to 10 users use the full core product at no cost.
• Evaluation: larger Jira sites can start a 30-day evaluation through the Atlassian Marketplace licensing flow.
• Active: the full product.
• Inactive: when the subscription is inactive the app shows “TestWarden is read-only: the subscription is inactive.” Your test data is preserved and stays readable; it becomes editable again when the subscription is reactivated in Atlassian Marketplace.

Billing, renewals and cancellations are handled by Atlassian Marketplace. See Pricing.

• “No changes received yet.” CI has not submitted evidence. Confirm App REST APIs are enabled for TestWarden in Atlassian Administration → Connected apps, the 3LO integration is authorized with both custom scopes, and your CI job targets the correct endpoint.
• Evidence rejected. Check that the payload names the exact repository, commit, build and environment, is under 256 KiB, has 1–20 observations, and contains no caller-supplied state, progressionAllowed or approval fields.
• A change stays HELD after a fix. Fresh evidence must be submitted for the new build. Evidence for an older build is marked superseded and the earlier HELD decision remains in history by design.
• VERIFYING turned into HELD with verification_run_overdue. The announced run did not deliver evidence inside its window. Re-run CI so it submits evidence for this build.
• I cannot archive, restore or migrate. These actions require the QA Admin role, derived from your Jira permissions.
• Jira issue created but not linked. Use Retry link on the step; the issue key is shown in the alert.
• Passed is disabled when finishing an execution. Resolve every Failed, Blocked and Not Run step first.
• The app is read-only. The subscription is inactive; reactivate it in Atlassian Marketplace.

Email [email protected] with your Jira site URL, the TestWarden app version and diagnostic reference from Help and feedback, what you expected, what happened and steps to reproduce. See Support for response targets and how critical issues are handled. Report security concerns separately to [email protected].