Test evidence for regulated releases

Two readers, one artifact

A test run produces evidence for two audiences who want different things from it.

The engineer wants to know what broke and where. A stack trace, a screenshot at the moment of failure, the locator that did not resolve, the network call that timed out. Depth of debugging evidence, as close to the code as possible.

The auditor wants to know something else entirely: that the release was tested against its requirements, that the testing actually happened as recorded, that nobody altered the record afterward, and that all of this can be demonstrated to a third party who was not in the room. Not depth. Provenance.

Most test tooling is built for the first reader. The second reader is usually served by exporting a report and hoping. This article is about what the second reader actually needs — drawn from the regulation that defines it most precisely — and why AI-driven test execution changes what has to be recorded.

What the regulation actually says

The clearest statement of what an audit-grade record looks like is in the US FDA's rule on electronic records, 21 CFR Part 11. It governs life sciences, not software testing generally, but it is the most precise definition in any regulation of what "evidence" means, and the properties it names are the ones every other regime borrows.

Section 11.10(e) requires the use of secure, computer-generated, time-stamped audit trails to independently record the date and time of operator entries and actions that create, modify, or delete electronic records. Record changes must not obscure previously recorded information. The audit trail must be retained at least as long as the records it describes and be available for agency review and copying.

Read that as a specification and it decomposes into five properties:

1. Computer-generated. The system writes the record, not the operator. A tester's notes are not an audit trail.
2. Time-stamped. Every entry carries when it happened, from a clock the operator does not control.
3. Independent. The record is produced separately from the action it describes, so the actor cannot shape it.
4. Non-obscuring. A change is recorded as a new entry. The prior state stays visible. Nothing is overwritten.
5. Retained and reviewable. Kept as long as the release it supports, and producible on request in a form a third party can read.

In practice, the audit trail must capture who performed an action, what action was performed, when it occurred, and why the change was made. Regulators expect audit trails to be reviewed periodically — failure to review them is itself a common inspection finding.

The regulation does not ask whether your tests passed. It asks whether you can prove what your tests did.

Why a passing suite is not evidence

Apply those five properties to a typical automated test run and the gaps are immediate.

A green dashboard is a summary, not a record. It says the suite passed; it does not say what the suite checked, against which requirement, or whether the tests that ran are the tests that were approved. A results file exported after the fact is operator-produced, not system-generated. A test that was repaired mid-cycle to make it pass has, from the auditor's perspective, obscured its own prior state — unless the repair was itself recorded as an entry with a reason.

None of this is a criticism of the tools. It reflects that testing tools were designed to help engineers find defects, and auditors were an afterthought served by a PDF export. That was survivable when a human wrote every test and every fix, because the human could be deposed. It stops being survivable when the tests are executed, healed, and interpreted by a system that makes decisions on its own.

What AI-driven execution adds to the problem

When an AI executor resolves a plain-English step to an element on a page, or substitutes a new element after the original moved, or reads an ambiguous instruction one way rather than another, it has made a decision that a human would previously have made and could previously have been asked about.

Those decisions are exactly what an auditor will want to see, because they are exactly where a test can drift from what was approved without anyone noticing. A self-healing tool that silently swaps one button for another has changed what the test checks. If that change is not recorded — who, what, when, why, with the prior state preserved — the test has failed property four, and the green result on the dashboard is not evidence of anything.

The question to ask any AI-driven tool is therefore not whether it records results. Every tool does. It is whether it records its own decisions with the five properties above: generated by the system, time-stamped, independent of the run, non-obscuring of prior resolutions, and producible in a form someone outside the QA team can read.

What that looks like in Spec2RunAI

Spec2RunAI, the test executor inside Spec2TestAI, was built on the assumption that the decision record is the product. Every element resolution, every healing substitution, and every interpretation of an ambiguous step is written to the run record by the system, time-stamped, with the candidate that was chosen and the basis for choosing it. Prior resolutions are not overwritten when a step heals; the new resolution is appended alongside the old. Each entry attaches to the business step as the tester wrote it, so the artifact reads without translation.

We describe this as the full AI decision audit trail, and we describe it that way deliberately: the five properties above are the design target. We do not claim Part 11 compliance for the executor — that is a determination made per system, per deployment, by the regulated party — and any vendor who claims it for a general-purpose tool is telling you something about their understanding of the regulation.

At a major financial regulator, this approach was associated with approximately 90% fewer requirements-phase defects. We report that as an engagement result, not an industry figure.

What to ask before a regulated release

Five questions, one per property. Any tool that cannot answer all five is producing results, not evidence.

• Does the system write the run record itself, or does a person assemble it afterward?
• Is every entry time-stamped from a source the operator does not control?
• When a test heals or is repaired, is the change recorded as an entry with a reason — and is the prior state still visible?
• Can the record be produced, in a form a non-engineer can read, for a release that shipped a year ago?
• For AI-driven execution specifically: does the record show what the system decided and why, or only what it did?

Our comparison of codeless test automation tools asks a version of the last question across six architectures, and concedes where the open frameworks do well on debugging depth. Depth is not the axis here. Provenance is.

Sources

• US Code of Federal Regulations, Title 21, Part 11, “Electronic Records; Electronic Signatures”, §11.10(e) — controls for closed systems; audit trail requirements. Text as published in the eCFR.
• Regulator outcome: AgileAI Labs engagement result. Not an industry figure; outcomes vary with existing requirements maturity and release cadence.