What test maintenance actually costs

The number nobody writes down

Ask a QA lead what their automation suite costs and you will hear a license figure, maybe a headcount. Ask what it costs to keep — the hours spent each release making yesterday's tests pass against today's build — and the answer is usually an estimate, offered with a shrug.

That is the line that decides whether a program finishes. Build cost is paid once. Maintenance is paid every release, forever, and it competes directly with the capacity that was supposed to go into new coverage. When repair consumes the sprint, coverage plateaus — not because anyone decided to stop, but because nothing is left to continue with.

This article separates maintenance into its parts, shows what the published research says about each, and gives a way to measure yours. It does not give a dollar figure, because no published dataset does, and we would rather say so than invent one.

Three things called "maintenance"

The word covers three activities with different causes, different costs, and different fixes. Budgeting them together is why the number is always fuzzy.

1. Locator repair

A test fails because the element it was looking for moved, was renamed, or was wrapped in a new container. The application still works. The test simply cannot find its way to the thing it was going to check.

This is the largest bucket by a wide margin. In a study of 300 versions of five web applications, over 70% of the 722 recorded test breakages traced to element locators rather than to application faults (Hammoudi, Rothermel & Tonella, IEEE ICST, 2016). Locator strategy changes the rate but does not remove it: the least robust strategy measured failed 83% of the time after a page change, and the most robust single strategy still failed 39% (Nass, Alégroth, Feldt, Leotta & Ricca, ACM TOSEM 32(3), 2023).

Each repair is trivial and manual — open the test, inspect the page, write a new locator. Minutes. The cost is the multiplier: a shared component change can invalidate hundreds at once.

2. Flakiness triage

A test passes and fails on the same build with no change to either the test or the application. Someone has to decide whether it is telling the truth this time.

The largest empirical study of UI-specific flakiness examined 3,516 commits across 7,037 repositories and isolated 235 distinct flaky UI tests. The single largest root cause — 45% of the sample — was asynchronous waiting: the test proceeded before a network resource, animation, or render had completed (Romano, Song, Grandhi, Yang & Wang, IEEE/ACM ICSE, 2021).

Triage is expensive in a way repair is not, because the cost is attention rather than typing. A team that stops trusting its suite starts re-running failures by reflex, and a suite that is re-run by reflex has stopped being a signal.

3. Genuine regression

The test fails because the application changed behavior and the test was right to notice. Of the three, this is the one that is actually testing. It is also, per the research above, the minority case.

Most of what a team calls test maintenance is not testing. It is restoring tests that were never wrong, so they can go back to checking things that did not change.

How to measure yours

Four numbers, one release cycle. Most teams have never collected them, and the first collection is usually the moment the maintenance conversation changes.

1. Total QA engineer-hours in the release. Everyone who touched the suite, all activities.
2. Hours spent on failed tests where the application was not at fault. Locator repairs, flake investigations that ended in "re-ran it, passed," timing adjustments. This is the maintenance tax.
3. Hours spent on failed tests where the application was at fault. Real regressions found and confirmed. This is what testing is for.
4. Hours spent writing new tests. The capacity that survived.

Divide line 2 by line 1. That ratio is what your automation program actually costs to keep, expressed in the unit that matters — the fraction of your team's time that goes to standing still.

Our ROI calculator models maintenance at 20–40% of build cost annually. We describe that as a floor rather than an estimate, because the measured repair costs in the locator literature run higher at any release cadence above quarterly. If your line-2 ratio comes out under 20%, you have an unusually stable suite or an unusually slow release cycle. If it comes out over 40%, you are not alone, and the calculator will accept your number in place of ours.

What reduces each bucket

Locator repair responds to two things: better locator strategy, which moves you from 83% toward 39% failure, and not holding locators in the test at all, which removes the mechanism. Spec2RunAI, the test executor inside Spec2TestAI, takes the second approach — steps stay in the tester's language, elements are resolved at run time, and proven resolutions are cached so replay does not repeat the work. A renamed button is not an event, because the test never named the button.

Flakiness triage responds to distinguishing "the test could not proceed" from "the test proceeded and the check failed." Async-wait flakiness is almost entirely the first kind, and a tool that reports it as such — rather than as a red result indistinguishable from a real regression — removes most of the triage cost. Ask any tool you are evaluating whether it separates those two outcomes in its results.

Genuine regression should not be reduced. It should be made cheaper to confirm: a failure attached to a business step with the evidence beside it takes minutes to triage; the same failure attached to a code line takes a developer.

What this does not tell you

None of the studies above produce a dollar figure, and we are not going to derive one from them. The ratios are the finding. What a 30% maintenance tax costs depends entirely on what the 100% is — a two-person team and a twelve-person team have the same ratio and very different bills. That is why the calculator asks for your rates rather than assuming them, and why the measurement above is in hours before it is in anything else.

Sources

• Hammoudi, M., Rothermel, G. & Tonella, P., “Why do Record/Replay Tests of Web Applications Break?”, IEEE ICST, 2016 — 300 application versions, 722 breakages; over 70% attributed to locators.
• Nass, M., Alégroth, E., Feldt, R., Leotta, M. & Ricca, F., locator robustness evaluation, ACM TOSEM 32(3), 2023 — absolute XPath 83% non-located; most robust single locator 39%.
• Romano, A., Song, Z., Grandhi, S., Yang, W. & Wang, W., “An Empirical Analysis of UI-based Flaky Tests”, IEEE/ACM ICSE, 2021 — 235 flaky UI tests from 3,516 commits across 7,037 repositories; asynchronous wait the largest root cause at 45%.
• The 20–40% maintenance ratio is a modeled floor used in the AgileAI Labs ROI calculator, not a published measurement; the calculator accepts a user-supplied value in its place.