QA Automation: What It Is and What It Costs to Keep

Every QA automation program starts the same way and a lot of them end the same way. Tests get written, coverage climbs, the dashboard looks good. Two years later the suite takes forty minutes, fails unpredictably, and half of it is skipped with a comment saying "flaky, fix later" dated eighteen months ago.

The interesting question was never how to automate a test. It is how to keep hundreds of them worth trusting.

What QA automation is

QA automation is using software to execute and evaluate tests that a person would otherwise run by hand. A script drives the application, checks the result, and reports pass or fail without anyone watching.

The reason to do it is not that machines are better testers. It is repetition. A human running the same regression pass for the two hundredth time is slow, expensive and, by the end, careless. A machine is none of those, and it can run on every commit rather than once a release.

What it does not replace is a person deciding what should be true. That is specification work, and it stays human.

What to automate

The test is repetition times consequence.

Automate: regression paths that must keep working, the core revenue journeys, anything run on every release, cross-browser checks, and data-heavy permutations a person would get bored doing.

Do not automate: one-off checks, anything requiring aesthetic judgment, exploratory testing (the point is that a person is looking for the unexpected), and features still changing shape weekly. A test written against a design that changes next sprint is work you will throw away and then do again.

The honest failure mode here is automating for coverage percentage rather than for consequence. A suite at 80% coverage that misses checkout is worse than one at 30% that does not.

Where the money actually goes

Three costs, and teams consistently mis-rank them.

Writing tests. Historically the visible cost, and the one that made automation feel expensive. Real, but never the largest.

Running them. Slow suites cost wall-clock time and CI minutes. Manageable with parallelism and sharding; hardware is cheap next to engineers.

Maintaining them. This is where programs die.

A test that finds a button by CSS class works until someone renames the class. One redesign against a few hundred selector-bound tests produces a wall of failures that have nothing to do with whether the product works. Somebody spends a week fixing them, or the noisy ones get disabled to unblock a release and nobody re-enables them.

Coverage then decays while the dashboard stays green, which is worse than having no suite, because it produces confidence that is not warranted.

Flakiness accelerates it. A test failing one run in twenty teaches the team to re-run rather than investigate. Once "just re-run it" is the reflex, the suite has stopped being a signal and is only a delay.

What changed

Two shifts, and it is worth separating them from general AI enthusiasm.

Authoring cost collapsed. When the agent writing a feature can also exercise it in a real browser and save the result, the afternoon per test stops deciding whether a test exists. That afternoon was the reason tests got dropped under deadline. See shift left testing.

Maintenance became addressable. The decay problem was never that tests were slow. It was that they were bound to selectors that change. A test expressed as intent, "click the Save button," can be re-resolved against the current page when the DOM moves rather than failing outright. That removes the largest maintenance category, though not all of it.

And one shift in the other direction, which is the honest part: agents refactor and rename far more aggressively than cautious humans. Selector churn goes up at exactly the moment test volume does, so a selector-bound suite in an agent-heavy codebase decays faster than it used to. The old approach is getting worse, not standing still.

What a durable program looks like

1. Cover consequence, not percentage. The journeys whose failure would make you roll back.
2. Write tests as intent, not selectors, or you are building the decay in on day one.
3. A smoke suite first, so a broken build fails in ninety seconds rather than forty minutes.
4. Treat flakiness as a defect. Quarantine, then fix or delete. Never "re-run and move on."
5. Gate merges, with the suite fast enough that nobody wants to bypass it.
6. Review generated tests. A test nobody read is a liability, because you cannot tell whether it asserts the right thing.

Shiplight addresses authoring and maintenance: the coding agent verifies a change in a real browser through MCP, and the successful walkthrough is committed to your repository as intent-based YAML. Tests transpile to Playwright and run in ordinary Playwright browsers alongside an existing suite, and a stale locator is re-resolved from intent with the heal surfaced as a reviewable diff rather than a silent rewrite. Web applications; native mobile is not in it.

Frequently Asked Questions

Related Reading

• End-to-end testing: the layer most of this concerns
• Smoke testing: the fast gate in front of it
• How to fix flaky tests: the thing that erodes trust
• Locators are a cache: the mechanism behind decay
• Quality engineering: the discipline around the suite