End-to-End Testing: What It Is and What It Costs

A unit test can pass while the feature is broken. That is not a criticism of unit tests, it is a description of what they measure: one piece behaving correctly in isolation, with everything around it replaced by stand-ins. Software fails between the pieces at least as often as inside them.

End-to-end testing is the layer that checks the assembled thing. This guide covers what an E2E test actually is, how it relates to the other layers, what it genuinely costs, and what changed about that cost recently.

What an end-to-end test is

An end-to-end test exercises a running application the way a user would, from the interface through to whatever the action was supposed to change. It clicks, types, waits and asserts, against a real deployment with real dependencies.

A concrete example. To test checkout, an E2E test opens the site, signs in, adds an item, enters payment details, submits, and confirms the order appears in the account history. No mocks, no shortcuts into the database, no calling the checkout function directly. If any layer is broken, the test fails.

That last property is both the point and the problem. It catches integration failures that no isolated test can, and it fails for more reasons, including reasons unrelated to your code.

The layers, and what each one buys

The received wisdom is a pyramid: many unit tests, fewer integration, fewest E2E, on the grounds that E2E is slow and expensive.

That advice was calibrated for a world where a human wrote and maintained every test. The shape is still sensible, but the reason for the specific proportions was economic rather than principled, and the economics moved. More on that below.

What E2E catches that nothing else does

Intent inversions. You asked for "sort newest first" and got oldest first. The code compiles, types check, unit tests pass because they assert the sort function sorts. Only something asserting the observed order catches it.

Integration failures. A component that renders correctly in isolation and breaks with real data shapes. A form that validates in tests and submits wrong in a browser.

Cross-browser divergence. CSS or JavaScript that works in one engine and breaks in another. Nothing below this layer opens a browser.

Silent removals. A refactor drops a null check, a rate limiter or an offline fallback, and the unit tests around that module were updated by the same agent that removed it. A test driving the running application is harder to accidentally satisfy. See regression risk in AI-generated code.

Real conditions. Authentication, live data, a specific browser state. Things that only exist in an assembled system.

The real cost, stated plainly

E2E testing has a reputation for being expensive. Worth being precise about where the expense actually is, because teams usually blame the wrong part.

Writing them is not the main cost. It never was. A test is an afternoon at most.

Running them is a manageable cost. Slow, yes, but parallelism and sharding handle it, and hardware is cheaper than engineers.

Maintaining them is where the money goes. This is the whole problem. A test that finds a button by CSS class works until someone renames the class. Multiply by several hundred tests and one redesign, and the suite fails for reasons unrelated to whether the product works.

What follows is predictable. The team disables the noisiest tests to unblock a release. Nobody re-enables them. Coverage decays while the dashboard stays green, which is worse than having no suite, because it produces confidence that is not warranted.

Flakiness compounds it. A test that fails one run in twenty teaches people to re-run rather than investigate. Once "just re-run it" is the reflex, the suite has stopped being a signal.

What changed

Two things, and they are worth separating from the general enthusiasm about AI.

Authoring cost approached zero. When the agent that writes a feature can also exercise it in a browser and save the result, the afternoon per test stops being the constraint. See shift left testing for why that historically decided whether tests got written at all.

Maintenance became addressable. The decay problem was never really about tests being slow; it was about tests being bound to selectors that change. A test expressed as intent, "click the Save button," can be re-resolved against the current page when the DOM moves, instead of failing. That does not eliminate maintenance, but it removes the largest category of it.

There is a third change pushing in the other direction, which is the honest part. Agents refactor and rename more aggressively than cautious humans do, so the rate of selector churn goes up at exactly the moment test volume does. A selector-bound suite in an agent-heavy codebase decays faster than it used to.

What a workable setup looks like

1. Cover the journeys that make money. Sign-up, sign-in, the core action, checkout. Not every path, the ones whose failure you would roll back for.
2. Write tests as intent, not selectors. Otherwise you are building the decay in from the start.
3. Run a small smoke suite first so an obviously broken build fails in ninety seconds rather than forty minutes.
4. Gate merges on the full suite, kept fast enough that nobody wants to bypass it.
5. Treat flakiness as a defect, not weather. One re-run culture and the signal is gone.

Shiplight covers authoring and maintenance: the coding agent verifies a change in a real browser through MCP, and the successful walkthrough is committed to your repository as intent-based YAML. Tests transpile to Playwright, so they run in ordinary Playwright browsers alongside an existing Playwright suite, and a stale locator is re-resolved from intent with the heal surfaced as a reviewable diff rather than a silent rewrite. Web applications; native mobile is not in it.

Frequently Asked Questions

Related Reading

• Smoke testing: the fast gate that runs first
• E2E vs integration testing: where the boundary sits
• Shift left testing: why authoring cost changed
• Locators are a cache: the mechanism behind suite decay
• How to fix flaky tests: the other thing that kills a suite