Platform / Enterprise Applications / Release drift model
What a vendor release does to an enterprise test suite. Measured rates, synthetic rows.
The charts on this page are computed from a synthetic dataset of 8,340 runs and 115,319 executed steps, generated to match rates measured on live Spec2RunAI runs against partner enterprise environments: a Guidewire PolicyCenter and an Insurity policy administration system, over nine months with two vendor releases each. Customer and partner agreements prevent publishing the run records themselves, so the rows are synthetic. The rates that shape them are not.
How to read this. Two kinds of number appear below. Rates — breakage before and after a release, recovery, verification coverage, replay, engagements per run, failure mix — were measured on real runs and carried into the model as its parameters. Counts — 8,340 runs, 1,559 selector breaks, 675 rewrites — are the size of the synthetic dataset and should not be read as production volumes.
Release dates in the model are representative, not the actual upgrade dates of the partner environments. Individual rows, test cases, and runs are generated and correspond to no customer record.
A vendor release multiplies step breakage by roughly four.
The view compares the share of executed steps that failed or were flagged for drift in the 14 days before each release with the 14 days after. Drift means the step passed but the wording it relied on no longer matches the page. The before and after rates are measured; the synthetic rows reproduce them around representative release dates.
Table view
| Window | Runs | Steps | Broken | Rate |
|---|---|---|---|---|
| Guidewire release 1, before | 228 | 3,309 | 26 | 0.79% |
| Guidewire release 1, after | 234 | 3,360 | 135 | 4.02% |
| Guidewire release 2, before | 228 | 3,187 | 35 | 1.10% |
| Guidewire release 2, after | 224 | 3,249 | 131 | 4.03% |
| Insurity release 1, before | 176 | 2,487 | 28 | 1.13% |
| Insurity release 1, after | 205 | 2,849 | 118 | 4.14% |
| Insurity release 2, before | 177 | 2,558 | 24 | 0.94% |
| Insurity release 2, after | 198 | 2,727 | 107 | 3.92% |
Most selector breaks are recovered without a person. The wording changes only with one.
Two different things can happen when a page changes under a step. The engine can find the element another way and carry on — that is recovery, and the stored test does not change. Or the wording in the test no longer describes the page, so the engine proposes a clearer wording and a person decides. The two are kept apart because they have different owners.
A passing run is not a verified run. The gap is measurable.
Verification coverage is the share of a run’s steps whose expected result was checked and met. A step that ran and passed with nothing confirming the outcome counts as not verified. In the measured runs, 23% of executed steps had no expected result written by the tester, which is the main reason coverage sits below 100%: the engine cannot verify what nobody specified.
Table view
| Coverage band | Guidewire runs | Insurity runs |
|---|---|---|
| 0–10% | 28 | 23 |
| 10–20% | 44 | 32 |
| 20–30% | 46 | 38 |
| 30–40% | 43 | 42 |
| 40–50% | 52 | 73 |
| 50–60% | 351 | 203 |
| 60–70% | 690 | 634 |
| 70–80% | 1,451 | 1,285 |
| 80–90% | 1,204 | 875 |
| 90–100% | 723 | 503 |
The first run is the expensive one. By the third, memory carries most of the steps.
Spec2RunAI bills AI engagements: four when a step is resolved for the first time, one when it replays from selector memory. The view follows 240 synthetic test cases through their first twenty runs, reproducing the measured first-run and steady-state rates.
Table view
| Run no. | Mean engagements | Memory replay |
|---|---|---|
| 1 | 15.77 | 0% |
| 2 | 3.83 | 73% |
| 3 | 2.93 | 78% |
| 4 | 2.85 | 79% |
| 5 | 3.14 | 78% |
| 6 | 3.1 | 79% |
| 7 | 3.01 | 78% |
| 8 | 3.07 | 78% |
| 9 | 3.12 | 77% |
| 10 | 3.07 | 78% |
| 11 | 3.18 | 78% |
| 12 | 3.05 | 77% |
| 13 | 2.94 | 78% |
| 14 | 3.13 | 79% |
| 15 | 2.95 | 78% |
| 16 | 3.21 | 78% |
| 17 | 3.26 | 76% |
| 18 | 3.31 | 77% |
| 19 | 3.3 | 77% |
| 20 | 3.33 | 77% |
A failure classified by cause goes to the right person.
Every failed run is classified as an application defect, a test defect, a data problem, or an environment problem, with a confidence score. The mix below reproduces the measured one: roughly half of failures are real defects.
How the model was built.
What was measured
Aggregate rates from live Spec2RunAI runs against two partner enterprise environments, January to October 2026: step breakage in the 14 days before and after a vendor release; the share of selector breaks recovered by the engine; the status mix of suggested rewrites; the distribution of per-run verification coverage; the share of steps without an expected result; memory-replay rate and AI engagements by run number; failed runs by classified cause; and Assertion Lock interventions per thousand steps. Rates were pooled across the two environments, which is why the two appear alike on this page.
What was generated
A seeded generator produced 240 synthetic test cases, 8,340 runs, 118,539 step rows (115,319 executed, 3,220 skipped), 94,243 selector-memory events, and four representative release dates, with outcomes drawn to match the measured rates. No row corresponds to a customer record. Counts on this page describe the synthetic dataset, not production volume.
Definitions used in every view
Executed step: a step that ran; skipped steps are excluded. Broken step: failed, or passed with drift flagged. Release window: 14 days either side of the date the environment moved to a new vendor release. Verified: expected result checked and met. Memory replay: step executed from stored selector memory with no model call. Engagement: one billed AI resolution unit; four on first resolution of a step, one per replay.
Why the rows are synthetic
The run records sit under customer and partner agreements that do not permit publication, including in anonymized form. Aggregate rates about Spec2RunAI’s own behavior are ours to report; the records behind them are not. Generating rows that reproduce the measured rates lets the distributions be shown without exposing any record.
What will not be published
Customer or carrier names, URLs, configuration details, step text, dataset values, screenshots, actual upgrade dates, or the run records themselves. Where a partner later permits it, a measured edition will replace this page with the same views computed directly from the records.
About this page.
Is this real customer data?
No row on this page is a customer record. The rates that shape the charts were measured on live Spec2RunAI runs against partner enterprise environments; customer and partner agreements prevent publishing the records, so the rows were generated to match the measured rates. Rates are measured; counts describe the synthetic dataset.
Why synthetic rows instead of the records?
Because the records belong to the environments they were run on. Aggregate rates about how Spec2RunAI behaves are ours to publish; a run against a partner’s policy administration system is not. Synthetic rows let the distributions be shown without exposing a single record.
Which numbers are measured and which are generated?
Measured: breakage rates before and after a release, the recovery share, the rewrite status mix, verification coverage, the share of steps without an expected result, replay rate, engagements by run number, failure cause mix, and the Assertion Lock rate. Generated: every individual row, and therefore every count (runs, steps, breaks, rewrites). Release dates in the model are representative, not the partners’ actual upgrade dates.
Can I get these numbers for my own application?
Yes. Send us ten of your manual test cases, untouched. We run them against your application and return the Run Evidence Report, which contains verification coverage, failure classification, and the AI provenance appendix for your runs.
Ten test cases. Untouched. Your application.
We run them as written and send back the Run Evidence Report: verification coverage, failures by cause, and the AI provenance appendix, for your runs.