Manual to automated testing: a migration guide

The problem is not teams with no automation

The premise behind most migration advice is that there is a large population of teams automating nothing. There isn't. In PractiTest's State of Testing 2024 survey, 9% of testers reported using no automation whatsoever — a minority, and a shrinking one.

The interesting number is at the other end. In the same survey, only 2–3% reported that automation had completely replaced manual testing, and the most commonly reported level of automation was up to 50% of previously manual work. Increasing automation was the single most cited priority, named by 30% of respondents.

Read those together and the shape of the problem changes. Almost everyone has started. Almost nobody has finished. The majority are sitting below half coverage, paying for an automation program and a manual regression cycle at the same time, and saying their top priority is to automate more.

The migration problem is not getting from zero to one. It is getting from partial to complete — and the thing blocking it is usually the same thing that blocked it the first time.

Why programs stall below half

Three causes account for most of it, and only one of them is about tooling.

1. The conversion tax is paid per test, forever

A manual test case is written for a person. It says things like “log in as a broker with an expired license and try to bind the policy.” Converting that into an automated test — whether into code, or into a vendor's supported command vocabulary — is a translation job, and it costs roughly the same for the five-hundredth test as it did for the first.

Teams budget for the first hundred, discover the cost is linear rather than front-loaded, and stop when the budget does. The tests that get automated are the ones automated early; the rest stay manual indefinitely. That is what a stall looks like from the inside.

2. Maintenance eats the capacity meant for expansion

Once a suite exists, someone has to keep it green. Research on why web tests break found that over 70% of breakages trace to element locators rather than to genuine application faults, across 300 application versions and 722 recorded breakages (Hammoudi, Rothermel & Tonella, ICST 2016).

Locator strategy compounds this. In a large comparative study of locator robustness, absolute XPath locators failed to find their target element in 83% of cases after change — the least robust strategy measured (Nass, Alégroth, Feldt, Leotta & Ricca, ACM TOSEM 32(3), 2023).

The practical effect is that the engineer who was going to automate the next fifty tests spends the sprint repairing the last two hundred. Coverage plateaus not because anyone decided to stop, but because there is no capacity left to continue.

3. The work sits with the wrong people

Automation is usually owned by engineers, while the domain knowledge — which edge cases matter, what “correct” means for a broker with an expired license — sits with testers and analysts. Every test therefore requires two people and a handoff. The queue forms at the engineer.

A sequence that works

Step 1 — Count what you actually have

Before choosing anything, count: total manual test cases, how many are run each regression cycle, how many cycles per year, and average steps per case. Nearly every cost model — including our own ROI calculator — is driven by those four numbers, and most teams have not written them down.

Then separate the suite into three buckets: tests run every cycle, tests run at release only, and tests nobody has run in a year. The third bucket is usually larger than expected and should not be migrated at all.

Step 2 — Decide who will own tests after a UI change

This is the question that determines the outcome, and it is not the same as “who authors tests.” Many tools make authoring accessible to non-engineers and quietly hand maintenance back to engineering the first time a selector moves.

Ask any candidate tool what happens when a button is renamed: who receives the failure, what they must understand to fix it, and whether the fix is reviewed. We wrote a longer version of this evaluation in our comparison of codeless test automation tools.

Step 3 — Migrate the highest-frequency tests first, not the easiest

The instinct is to start with simple tests. The economics say start with frequent ones. A test run every cycle returns its migration cost far faster than a complex test run twice a year, regardless of which is easier to convert.

Step 4 — Keep the manual suite running until coverage is proven

Run both in parallel for at least two full cycles and compare results. Any test that passes automated and fails manually — or vice versa — is telling you something about the migration, not about the application. Retiring the manual cycle before that comparison is how teams discover, six months later, that a green suite stopped testing anything.

Step 5 — Measure the thing you were trying to fix

Coverage percentage is the wrong metric because it rises whether or not the tests are meaningful. Track instead: cycle time for a full regression pass, engineer-hours per release spent on test repair, and defects found by automation versus found downstream. If repair hours are not falling, the program is heading for the same plateau as last time.

Where the conversion tax can be avoided entirely

Steps 1 through 5 apply whichever tool you pick. But the largest single line in most migrations — Step 1's conversion cost — exists because the target tool requires tests in a particular form.

Spec2RunAI, the test executor inside Spec2TestAI, runs existing manual test scripts as written. No required keywords, no reformatting, no restructuring into a supported vocabulary. Business jargon and multi-action lines are handled as the tester wrote them; expected results are optional and per step; ambiguous steps are flagged, rewritten, and offered for one-click acceptance. Proven on real customer scripts.

That does not make the other four steps unnecessary. You still need to count what you have, decide who owns maintenance, sequence by frequency, and run in parallel before retiring anything. It removes one line from the budget — usually the largest one, and the one that caused the last stall.

If your suite runs weekly or nightly and maintenance is consuming capacity you would rather spend elsewhere, the three-year model is editable with your own numbers. If you run a very large suite many times a day, per-run pricing may not beat an in-house team, and the calculator will show you that too.

Sources

• PractiTest, State of Testing Report 2024 — 9% of testers report no automation; 2–3% report automation has fully replaced manual testing; most common automation level up to 50%; increasing automation the top priority at 30%.
• Hammoudi, M., Rothermel, G. & Tonella, P., “Why do Record/Replay Tests of Web Applications Break?”, IEEE ICST, 2016 — 300 application versions, 722 breakages.
• Nass, M., Alégroth, E., Feldt, R., Leotta, M. & Ricca, F., locator robustness evaluation, ACM TOSEM 32(3), 2023.