Why tests break: a look at locator strategy

The failure mode is not what most teams budget for

Ask a team why their automated suite is unreliable and the answer is usually about the application: timing, async loading, flaky environments. The published evidence points somewhere narrower and more fixable.

In a study of why record and replay tests of web applications break, researchers examined 300 versions of five applications and cataloged 722 individual breakages. Over 70% of them traced to element locators rather than to genuine application faults (Hammoudi, Rothermel & Tonella, IEEE ICST, 2016).

The majority of test failures are not the application telling you something. They are the test failing to find a thing that is still there.

Not all locators fail equally

If locators cause most breakages, the obvious next question is whether the choice of locator strategy changes the outcome. It changes it enormously.

A large comparative evaluation of locator robustness measured how often different strategies could still find their target element after the page changed. Absolute XPath was the least robust of those measured, failing to locate the element in 83% of cases. The most robust single-locator strategy in the same evaluation, an algorithmically generated relative XPath, failed in 39% of cases (Nass, Alégroth, Feldt, Leotta & Ricca, ACM TOSEM 32(3), 2023).

A separate comparative analysis of ID-based and XPath-based locators found 26% of XPath locators breaking across the versions studied, and 19% of cases in which neither an ID-based nor an XPath-based locator resolved the element.

Read those together and two things follow. First, locator strategy is the single largest controllable variable in suite stability — the gap between the best and worst measured strategies is more than a factor of two. Second, and less comfortably: even the best strategy measured still failed roughly two times in five. There is no locator strategy that makes this problem go away.

What this means for maintenance budgets

Most cost models for test automation treat maintenance as a percentage of build cost — 20% to 40% a year is a common range, and one we use as a floor in our own ROI model. What the locator research adds is a mechanism for where that percentage comes from.

A UI change that a user would not notice — a renamed class, a button moved inside a new container, a form field wrapped in a toggle — invalidates a locator without invalidating the test. The test still describes something true about the application. It simply cannot find its way to the thing it was going to check.

The repair is usually trivial and always manual: someone opens the test, inspects the page, and writes a new locator. It takes minutes. The problem is the multiplier. A design system change that touches a shared component can invalidate hundreds of locators at once, and the resulting sprint of repair work is why automation programs stall below half coverage while the manual regression cycle continues.

Three responses, in ascending order of effectiveness

1. Improve the locator strategy

Prefer stable identifiers over structural paths. Add dedicated test attributes to the application, so locators depend on something the team controls rather than on markup that changes for visual reasons. This is real and worthwhile: it moves you from the worst measured strategy toward the best.

The limitation is the 39% figure above. Even the most robust strategy measured still failed in a substantial minority of cases, so this reduces the repair queue rather than removing it.

2. Add self-healing

Most modern platforms can substitute an alternative element when the original locator fails. This removes much of the manual repair work, and it is the reason self-healing has become a standard feature rather than a differentiator.

The question worth asking any healing implementation is what it optimizes for. Healing that finds the element most similar to the one that vanished will turn a red test green — but a test that quietly starts checking the wrong element is worse than one that fails, because nothing tells you it happened. Ask whether a candidate is validated against the step's intent before it is accepted, and what record is kept of the substitution.

3. Do not hold locators in the test at all

The structural version of the fix is to keep the test as a description of intent and resolve the element at run time. A step that reads place the order has nothing in it to break when a button is renamed, because it never named the button.

This is the approach Spec2RunAI, the test executor inside Spec2TestAI, takes. Steps stay in the language the tester wrote them in; elements are resolved when the test runs; proven resolutions are cached so that subsequent runs replay without repeating the work, and the cache is refreshed when the page changes underneath it. Every resolution and every substitution is recorded, so a reviewer can see what was chosen and why.

That is not a claim that the approach is free of failure modes. Resolving elements at run time introduces a dependency on the model doing the resolution, which is why the decision record matters and why cached replay matters for cost. It does mean that a renamed button is not an event.

The practical test when evaluating a tool

Rename a button in a staging build and run the suite. Then ask three questions: how many tests failed, how many were repaired without human involvement, and what record exists of each repair. The first number tells you about locator strategy, the second about healing, and the third about whether you could explain any of it to an auditor.

We wrote the longer version of this evaluation, across six different architectures, in our comparison of codeless test automation tools.

Sources

• Hammoudi, M., Rothermel, G. & Tonella, P., “Why do Record/Replay Tests of Web Applications Break?”, IEEE ICST, 2016 — 300 application versions, 722 breakages; over 70% attributed to locators.
• Nass, M., Alégroth, E., Feldt, R., Leotta, M. & Ricca, F., locator robustness evaluation, ACM TOSEM 32(3), 2023 — absolute XPath 83% non-located; most robust single locator 39%.
• Comparative analysis of ID-based and XPath-based locators — 26% of XPath locators broken; 19% of cases with neither resolving.