7 min read
The cost of self-healing QA (real numbers)
I ran AI-driven self-healing on 1000+ tests for a global pet care company. This is what it cost and what it saved.
The pitch for AI-driven QA is that an agent watches your failing tests, figures out that a selector drifted, auto-patches the test, and saves you engineering time.
The pitch leaves out the part where agents also make mistakes, cost money per run, and occasionally patch a test so that it still passes but no longer tests anything useful.
I’ve been running this for about a year for a global pet care company across six apps and roughly 1000 Playwright tests. This post covers the costs, the savings and when I think it’s worth doing.
The baseline (before)
Each engineer on the QA team was spending roughly 8 hours a week on maintenance toil: selector updates, flake investigations, re-running builds to confirm a fix. Across a team of 4, that’s 32 hours a week, close to a full FTE spent just on keeping existing tests green.
This is how the failure modes broke down:
- 52% of failures: selector drift (DOM change, new class names, dynamic IDs changing)
- 21% of failures: timing / race condition flakes
- 15% of failures: real regressions (the thing we wanted to catch)
- 12% of failures: test bugs (bad assertion, wrong data)
Most of the work was on the 52% selector drift, which is the least intellectually interesting category.
What the agents replaced
The agents handle three specific interventions:
- Selector self-healing: when a test fails on an element-not-found error, the agent pulls the current DOM, finds the closest match to the old selector, and submits a PR with the fix.
- Flake quarantine: the agent re-runs intermittent failures 3 times, marks stably-failing as real, stably-passing as noise, and submits a retry annotation PR for the noise category.
- Test authoring from tickets: when a ticket hits “Ready for QA,” the agent drafts a Playwright test that matches the acceptance criteria.
The cost side
These are the monthly costs at current usage:
| Item | Monthly |
|---|---|
| LLM API calls (OpenAI + Claude mixed) | ~$180 |
| Agent infra (CF Workers + Lambda) | ~$25 |
| Storage (test artifacts, DOM snapshots) | ~$15 |
| Monitoring / observability | ~$20 |
| Total | ~$240 |
That’s about three grand a year to run the fleet.
The savings side
Of the 8 hours/week/engineer baseline, I estimate the agents are reclaiming roughly 5 hours/week/engineer. They don’t reclaim all of it, because novel failures still need human judgment and PR review of agent output takes time too.
Five hours × 4 engineers × 48 weeks = 960 engineer-hours per year.
At a fully-loaded cost of ~$100/hour (mid-range for senior QA), that’s ~$96,000/year in reclaimed time.
That leaves ~$93,000/year net positive, roughly a 32× return on the AI spend.
Two big problems
1. Trust calibration takes months
For the first three months, the team did not trust agent-authored PRs. Reviews were slow and defensive. We shipped the same agent fixes humans were already making, and the humans double-checked everything. Net velocity was roughly break-even.
By month four, the team had internalized which patterns the agent reliably got right (simple selector updates, text matcher changes, waiting on attribute changes) versus which it got wrong (anything involving state, timing, multi-step flows). Reviews got fast for the reliable patterns, and velocity picked up sharply.
The savings don’t compound until the humans have calibrated. If you budget 0 months of trust-building, the agents lose you money.
2. Silent patch drift
This is the scariest category. The agent notices a test is failing, finds a “close enough” selector and patches it, so the test passes while validating something subtly different than intended.
For example, the agent finds a button with the same text but in a different modal. The test passes, but it’s no longer testing the flow you wrote it for.
To mitigate this, every agent-authored PR has to explain what it thinks changed and what it thinks the test still validates. A human reads that claim and agrees or rejects. You cannot ship this without review, and if you allow auto-merge, you’ll eventually get a test suite that is 100% green and 40% wrong.
When it’s worth doing
Based on the math, self-healing QA pays off when you have:
- At least ~500 automated tests that currently see drift-driven maintenance
- Engineers who can review agent PRs competently (which means a technical QA team)
- A CI pipeline observable enough for an agent to read, reason about, and act on (structured test output, flake history, DOM snapshots on failure)
- Tolerance for a 2-3 month calibration period before savings show
It does not pay off when:
- Your suite is small (<200 tests); the absolute dollar savings won’t exceed the API cost + review overhead
- Your test suite is mostly flaky for architectural reasons (tests that need data setup, third-party services, test environment instability); those aren’t selector problems, and agents can’t fix them
- You don’t have a human review gate and plan to turn on auto-merge to save more time
Summary
Self-healing QA is a targeted replacement of mechanical work, budgeted in dollars and hours, with real costs and limits. At the scale we run it (1000+ tests, 4 engineers), the math is unambiguously positive, roughly a 32× return. At smaller scales it’s a wash, and without proper CI observability it’s negative.
The numbers here show AI making a good QA team substantially more productive.
If you’re running the math on AI-driven QA for your team, let’s talk. I’ll tell you whether it’s worth it at your scale.