Reports — run evidence
Every finished run has an evidence report: a printable record of what the engine did at each step, what it checked, and what the screen showed. It is written for the people who were not watching the run — a QA lead reviewing it, a business owner signing off UAT, an auditor asking about it months later.
Opening a report
| From | How |
|---|---|
| Reports in the sidebar | Lists finished runs, newest first, with their result, pass count and duration. Search by run ID, suite or project, filter by result, and use Load more for older runs. Click Open report. |
| Any run page | Click Evidence report, beside the card and table view buttons. It opens the report for the run you are looking at. |
Runs still in progress are not listed; they appear once they finish. The list shows each run's pass count, and a report's verification coverage appears when you open it.
What a report contains
| Section | What it shows |
|---|---|
| Run details | Run ID, source (suite, or project and folder), target, who started it, when, duration (with how much of it was the application's own loading time), engine, browser, number of test cases, AI engagements used, whether Autonomous Tuning was on, the date the screenshots are kept until, and which earlier run it is compared with. |
| Case results | How many test cases passed, labeled Clean, Tuned or Failed (see below). |
| Verification coverage | How many steps actually had their expected result checked. This is the most important number in the report. |
| For reviewers | Findings raised automatically from the run's evidence — the things a reviewer should look at first. |
| One section per test case | Every step with its expected result, what the engine observed, a screenshot, and the result. |
| Appendix A | AI provenance and integrity: Assertion Lock, script fingerprints, and every rewrite or drift suggestion. |
| Appendix B | Developer-ready defect reports for failed test cases, when failure triage wrote one. |
| Sign-off | Who executed the run, with blank lines for QA review and business sign-off. |
The report also covers everything the earlier Suite report showed (comparison with the previous run, stability, failure triage and a Markdown export), so a run now has one report.
Verification coverage
Every step in the report is in one of four states:
| State | Meaning |
|---|---|
| Verified | The step had an expected result, the engine checked it, and it was met. |
| Not verified | The step ran and passed, but nothing confirmed its expected result. There are three reasons, and the report says which: the step has no expected result; the verifier could not complete its check (for example, the AI call timed out); or the run is from before engine 2.10.131 and recorded no verification for that step. |
| Failed | The action failed, or its expected result was not met. The report shows the error. |
| Not run | Steps after a failure that never executed. |
For reviewers
The report lists findings in priority order, raised from the run itself:
- Failures — which step failed, and the error.
- Regressions and fixes — test cases that passed on the suite's previous run and fail now, or the reverse.
- Unstable test cases — below 80% stable over the suite's recent runs.
- Steps with no expected result — named, so you can add them.
- Verifier failures — steps whose check could not be completed.
- Steps with no verification record — only on runs from before engine 2.10.131.
- Steps run on rewritten wording — see Appendix A.
- Drifted steps — passing steps whose wording no longer matches the page.
- Application waiting — when the application's own loading time was 20 seconds or more, with the two longest waits.
When there is nothing to flag, the report says so.
Comparison with previous runs
For a suite run, the report compares each test case with the same suite's earlier runs. Runs started from the project tree or the repository are not compared, and the report says so under Compared with.
| Marker | Meaning |
|---|---|
| Regressed · was PASS | Passed on the suite's previous run and fails now. Something changed: the application, the test data or the environment. |
| Fixed · was FAIL | Failed on the previous run and passes now. |
| New in suite | Not in the previous run. |
| 50% stable · 3 runs | How often the verdict stayed the same between consecutive runs, over this run and up to two before it. 100% means it never changed. Below 80% is flagged for reviewers: a result that changes with nothing to explain it usually means the step wording is ambiguous. |
Failure triage
Each failed test case shows its triage above the steps: the classification (app defect, test defect, environment or data), the confidence, what happened, the suggested action and the evidence it was based on. When triage wrote a developer-ready defect report, it is collected in Appendix B, ready to paste into your defect tracker.
Reading a step
| Column | What it shows |
|---|---|
| Step and expected result | The step exactly as written, and its expected result — or a note that none was written. |
| What the engine observed | For a checked step, the verifier's own description of what the page showed. For a failed step, the error. When a typed or selected value was confirmed by reading the field directly, the report says so: that check needed no AI call. |
| Evidence | The step's screenshot. Click it to enlarge. On typing and selection steps, the control the engine acted on is outlined. After the wait means the screenshot was retaken once a loading overlay cleared, so it shows the page the step was judged on. |
| Result | The step's state, its duration, and any time spent waiting for the application. |
Where the engine did something worth knowing, an ENGINE note appears under the observation. It covers selector memory hits, dropdown handling, the selector guard rejecting a wrong element, vision or coordinate clicks, waiting for an overlay before acting, verification-only steps, self-healing, Autonomous Tuning rewrites and drift.
Case labels
| Label | Meaning |
|---|---|
| Clean | Passed on the steps as written. |
| Tuned | Passed after a step was reworded: Autonomous Tuning rewrote a failed step and re-ran it, or self-healing passed the test case with a suggested permanent fix. Your expected results were not changed. |
| Failed / Error | As on the run page. |
| N drifted | A badge, not a verdict: that many passing steps quote wording the page no longer has. See Autonomous Tuning. |
These match the verdicts on the Autonomous Tuning page, so a run reads the same in both places.
Appendix A — AI provenance and integrity
- Assertion Lock. The AI may change how a step is worded; it never changes what the step verifies. Every expected result in the report appears exactly as it was evaluated.
- Script fingerprint. A code (SHA-256) identifying each test case exactly as written, taken before step 1. If anyone edits the test case afterwards, its fingerprint changes, so the report proves which version ran. Recorded from engine 2.10.131 on.
- Rewrites. For each step Autonomous Tuning rewrote and re-ran, and each self-healing fix, the report shows the original wording, the wording that ran, the semantic check, and the unchanged expected result. The stored test case changes only when someone approves the rewrite in Autonomous Tuning, or when your organization has automatic application turned on.
- Drifted wording. Suggested wording for steps that passed, marked Verified rewrite or Unverified exactly as on the Autonomous Tuning page (see Verified Rewrite). These are suggestions only and were not applied in this run.
Screenshots, sign-off and PDF
The toolbar above the report controls what goes into it:
| Control | Use |
|---|---|
| Screenshots | All steps, Not verified and failed (the steps a reviewer needs to see), or None for a shorter document. |
| Sign-off block | Adds the executed-by panel and signature lines for QA review and business sign-off. |
| Download PDF | Opens your browser's print dialog. Choose Save as PDF. Only the report is printed; the sidebar and toolbar are left out, and a step is never split across two pages. |
| Open run | Jumps to the run page for the full detail and the AI Decision Inspector. |
| Copy Markdown / Download .md | The same report as Markdown text, for a Jira ticket, a Slack thread or a pull request. It carries the results, coverage, findings, a step table for each test case and the defect reports, but not the screenshots. |