How to evaluate self-healing before you trust it

Every tool says it. Almost nobody defines it.

Self-healing has become a checkbox. A vendor demo renames a button, the test goes red, something happens, the test goes green, and the room nods. What happened in between is the entire product, and it is almost never shown.

This piece is an evaluation guide. It defines what healing has to do to be trustworthy, describes the failure mode that a passing demo hides, and gives a test you can run in an afternoon against any tool that claims the capability — including ours.

Why healing exists

The published evidence is consistent: over 70% of web test breakages trace to element locators rather than to application faults, across 300 application versions and 722 recorded breakages (Hammoudi, Rothermel & Tonella, IEEE ICST, 2016). The test was right; it just could not find its way to the thing it was going to check. Healing is the industry's answer to that — when the original element cannot be found, find a replacement and carry on.

That is a reasonable answer to a real problem. The question is what "find a replacement" means, because there are two very different things it can mean, and only one of them is testing.

The failure mode a green result hides

A healing implementation that optimizes for getting the test to pass will find the element most similar to the one that vanished — by position, by attributes, by visual appearance — and substitute it. Most of the time that is the right element. Some of the time it is a different button that happens to sit where the old one sat.

When that happens, the test goes green. The check it performs is now against something other than what the tester intended. Nothing in the result distinguishes this from a legitimate pass, because from the tool's perspective the step succeeded. The suite has quietly started testing something else.

A test that fails loudly is a cost. A test that passes against the wrong element is a liability, because it has stopped being a test without telling anyone.

This is the property that separates healing implementations, and it is invisible in a demo. The demo shows one rename, one heal, one green. It cannot show the hundredth heal on a real suite where the substitution was wrong and nobody looked.

Four properties of healing you can trust

1. Candidates are validated against intent, not similarity

The question a healing step has to answer is not "which element looks most like the one that disappeared?" It is "which element, if any, fulfills what the tester wrote?" A step that says submit the claim should heal to an element that submits a claim, and refuse to heal to a "Save draft" button that happens to occupy the same position. That requires the tool to understand the step, not just the DOM.

2. Healing is bounded and escalates

A tool that will substitute anything to avoid a failure is not healing; it is suppressing. Trustworthy healing tries a small number of alternatives in order of confidence, and if none is validated against intent, it stops and reports the step as unresolvable. Ask how many alternatives are tried and what happens when they run out. "It always finds something" is the wrong answer.

3. Every substitution is recorded with its basis

When a heal occurs, the record should show the original resolution, the new resolution, the candidates that were considered, and why the chosen one was accepted. The prior resolution must remain visible — appended to, not overwritten. This is what makes a healed test reviewable: someone can look at the substitution later and confirm it was right, or catch that it wasn't.

4. The application being broken is reported as a bug, not healed around

If the "Submit" button is genuinely gone because a deploy removed it, the correct outcome is a failed test and a filed defect. A tool that heals to the next-best element has just concealed a regression. Ask directly: when the element is missing because the application is broken, what does the tool do?

The afternoon test

Take a staging build. Make three changes, one at a time, and run the suite after each.

1. Rename a button. Same function, new label, new id. This is the demo case. Every tool should heal it. What you are checking is the record: does it show what was substituted and why?
2. Swap two buttons' positions. "Submit" and "Cancel" trade places; labels and ids stay the same. A similarity-based healer keyed on position will heal to the wrong one. An intent-validated healer will not need to heal at all, because the labels still match.
3. Remove the button entirely. Delete the element the step needs. The correct result is a failure with a clear reason. A tool that goes green here has healed around a broken application, and you have just learned what it will do in production.

Then ask for the run record for all three and hand it to someone who was not in the room. If they can tell from the record alone which change was made and what the tool decided, the healing is auditable. If they can only see pass and fail, it isn't.

How Spec2RunAI approaches this

Spec2RunAI, the test executor inside Spec2TestAI, resolves elements from the plain-English step rather than from a stored locator, so a rename is not a healing event — there was never a locator to break. Where resolution does fail against a cached result, it tries up to three alternatives in escalating order, each validated against the step's intent before acceptance, and stops when validation fails. Every resolution and substitution is written to the run record with the candidates considered and the basis for the choice, and prior resolutions are preserved.

We describe this as healing with integrity, and the afternoon test above is the one we would ask you to run. Test 3 in particular: the correct result is a failure, and we would rather you see that than a green result you cannot trust.

What this does not cover

Healing addresses locator fragility. It does nothing for asynchronous-wait flakiness, which the largest empirical study of UI-specific flaky tests found to be the single largest root cause at 45% of 235 tests examined (Romano, Song, Grandhi, Yang & Wang, IEEE/ACM ICSE, 2021). That is a separate failure mode with a separate fix, and a tool that conflates the two under "self-healing" is describing something broader than the term supports.

Sources

• Hammoudi, M., Rothermel, G. & Tonella, P., “Why do Record/Replay Tests of Web Applications Break?”, IEEE ICST, 2016 — 300 application versions, 722 breakages; over 70% attributed to locators.
• Romano, A., Song, Z., Grandhi, S., Yang, W. & Wang, W., “An Empirical Analysis of UI-based Flaky Tests”, IEEE/ACM ICSE, 2021 — 235 flaky UI tests; asynchronous wait the largest root cause at 45%.