A passing run is not a verified run

What a result is, by definition

The ISTQB glossary defines an expected result as "the predicted observable behavior of a component or system executing under specified conditions, based on its specification or another source," and an actual result as "the behavior produced/observed when a component or system is tested." Pass/fail criteria are "decision rules used to determine whether a test item (function) or feature has passed or failed a test."

Read those three together and a pass has a structure. Something was predicted. Something was observed. A rule compared them. Remove the prediction and there is nothing to compare against; the step can complete, but it cannot pass in the glossary's sense. It can only not fail.

ISO/IEC/IEEE 29119-3:2021, the test documentation standard, keeps the pieces apart for the same reason. Among the documents it specifies for dynamic testing are actual results, test result, a test execution log, and a test incident report — four records, not one. The log says what happened. The result is a judgment about what happened, and the judgment needs the expected result to exist.

Dashboards, and the habits that grow around them, collapse the four into a color. Green means the run reached the end without an error being raised. Whether anything along the way was checked against a prediction is not what the color encodes, and in a suite that has been green for six months, nobody remembers to ask.

Where the unchecked steps come from

This is not a story about careless testers. Unchecked steps arrive through four ordinary routes.

The expected result lives at the end. A manual test case written by someone who knows the business often reads: log in as a broker with an expired license, open the policy, try to bind it — it should be blocked. One prediction, at the end, covering the whole path. The first steps are actions. Nobody wrote "the policy page should appear" because it was obvious. When that test is automated, the action steps run and complete, and only the last one is verified. The test is still a good test. It is a good test with one checked step.

Actions carry expectations nobody wrote down. "Click Add to Cart" implies the cart count should change. "Wait for the page to load" implies a particular page. An engine that clicks and moves on has done what it was told; it has not confirmed the implication, and if the click landed on the wrong control and the next step happened to work anyway, the run stays green.

The check could not complete. The expected result was written, but the verifier timed out, the element it needed was hidden behind an overlay, or the screen changed while the check was running. Whether that step is reported as failed, as unverified, or as passed depends entirely on how the engine is built. An engine that treats "the check did not throw" as "the check passed" reports it green.

In scripted suites, the assertion is the last line. A Playwright or Selenium test typically performs a sequence of actions and asserts at the end. That is a deliberate design, not a flaw, and the framework's own report is honest about it: assertions passed, and there were N of them. The question is whether anyone downstream reads the count, or only the color.

Why AI execution raises the stakes

When a script runs, a step without an assertion is inert. It does what it does. When an AI engine runs a plain-English step, it has to decide what "done" looks like — which control matched "the Continue button," whether the page that appeared was the one the step implied. The engine's decision that a step succeeded can be right, and usually is. But it is the engine's judgment standing in for a prediction the tester never wrote, and a reviewer cannot tell the two apart from a green cell.

The FDA's final guidance on Computer Software Assurance (September 2025) describes what an appropriate record of assurance activities includes: the intended use of the function, the result of the risk-based analysis, a description of the testing conducted, the issues found, "a conclusion statement declaring acceptability of the software for its intended use," and who performed the testing and when. For scripted testing it expects the result for each test case. A conclusion statement declaring acceptability is a claim about what was verified. If the record cannot distinguish steps that were checked from steps that merely ran, the conclusion is resting on something the record does not contain.

That is the same gap our earlier piece on regulated releases found in 21 CFR Part 11: a green dashboard is a summary, not a record. Verification coverage is the specific, countable version of that gap.

Four states, one number

The fix is to stop letting "pass" mean two things. Every executed step is in one of four states:

• Verified — an expected result was written, the engine checked it, and it was met.
• Not verified — the step ran and completed, but nothing confirmed an outcome: no expected result was written, or the check could not complete.
• Failed — an expected result was checked and not met, or the action itself could not be performed.
• Not run — the step was never reached, usually because an earlier step failed.

Verification coverage is the share of steps in the first state. It is not a quality score for the application. It is a statement about the evidence: of everything that ran, this much was actually confirmed. A run can be 100% passed and 40% verified, and both numbers are true.

Two consequences follow. First, "Not verified" is a finding about the test, not about the engine or the application. The remedy is to write the expected result — on the steps that carry the risk, not on every click. Second, a reviewer can now decide deliberately. A smoke test at 30% verified may be fine. UAT evidence for a claims workflow at 30% verified is not, and the number says so before anyone signs.

A worked example

On 27 September 2026 we ran a nine-step checkout test against a public demo store: browse, add to cart, apply a promo code, enter shipping and payment, place the order. It passed, 9 of 9 steps, in real Microsoft Edge on Windows 10. The evidence report scored it 56% verified.

Four of the nine steps had no expected result written — the click on the collection button, the wait for the catalog page, the add-to-cart click, the promo code entry. They ran. Nothing confirmed what they produced. The remaining five, including "the cart page should appear" and "the Thank you for your order screen should appear," were checked and met. The report's first finding for reviewers read: "4 steps had no expected result. They ran, but nothing was checked. Add expected results before using this run as sign-off evidence."

That is the correct outcome. The test passed, the application worked, and the report declined to let a 56% verified run be presented as a fully verified one. Ten minutes of adding expected results to the four steps turns it into evidence. Without the number, nobody would have known the ten minutes were owed.

What to ask before you trust a green result

These questions work against any tool, including ours, and can be answered from a sample report rather than a demo.

• For a passing run, how many steps had an expected result that was actually checked? Is that count on the report, or does someone have to derive it?
• When a step has no expected result, what does the report call it — passed, or something more honest?
• When a check cannot complete — timeout, hidden element, screen changed — is the step failed, unverified, or passed?
• Where the engine inferred what success looked like, does the report say so, step by step?
• Can a reviewer who did not watch the run tell, from the record alone, which steps to trust?

A vendor who can answer all five from a printed report has thought about the second reader. A vendor who answers with a dashboard tour has not yet.

What that looks like in Spec2RunAI

Spec2RunAI, the test executor inside Spec2TestAI, puts verification coverage at the top of every Run Evidence Report: the share of steps verified, the count in each of the four states, and, for each unverified step, the reason — no expected result written, or the verifier could not complete. Steps with no expected result are listed by name in the findings for reviewers. Every expected result appears in the report exactly as it was evaluated, under Assertion Lock, so the number cannot be improved by the engine quietly rewriting what a step was supposed to confirm. The report exports to PDF with signature lines, or to Markdown for a tracker.

None of that makes a test better. It makes the evidence honest about how good the test was — which is the property a sign-off needs, and the one a green result has never carried on its own.

Sources

• ISTQB, Standard Glossary of Terms Used in Software Testing, version 3.2 — definitions of "expected result," "actual result," and "pass/fail criteria."
• ISO/IEC/IEEE 29119-3:2021, Software and systems engineering — Software testing — Part 3: Test documentation, edition 2, October 2021 — dynamic test process documentation, including actual results, test result, test execution log, and test incident report.
• US FDA, Computer Software Assurance for Production and Quality Management System Software, final guidance for industry and FDA staff, issued September 24, 2025 (current version dated February 3, 2026); supersedes Section 6 of General Principles of Software Validation (2002) — contents of the record of assurance activities.
• US Code of Federal Regulations, Title 21, Part 11, §11.10(e) — audit trail requirements, as discussed in Test evidence for regulated releases.
• Worked example: AgileAI Labs run against the public Ember demo store, 27 September 2026, Microsoft Edge 153 on Windows 10 via BrowserStack; 9 of 9 steps passed, 5 of 9 verified.