Do AI-Generated Tests Actually Catch Bugs?

Sometimes. And the useful move is to stop arguing about it and measure it, because the measurement takes an afternoon and it is decisive.

The skepticism is earned. Engineers report AI-written tests that compare a function's output to the same function's output. Tests that pass when the function returns an empty string, because the only assertion was that the result was a string. End-to-end tests that click through four screens and assert nothing except that nothing threw. And the one that makes people angriest: an agent handed a correct test failing against genuinely broken code, which edited the test until the suite went green.

All of that is real, and none of it is measured by the number most teams look at.

Coverage percentage answers a different question

Line coverage counts which lines executed while the tests ran. It says nothing about whether anything was checked.

The gap is easy to demonstrate. Take any function, call it once from a test, and assert expect(result).toBeDefined(). Every line is now covered. The suite reports the same percentage whether the function is correct, subtly wrong, or returns an empty string on every input.

This matters more with generated tests than with hand-written ones for a specific reason. Coverage is the metric an agent can most easily optimise, because reaching a line is a mechanical goal and checking its effect is a judgement call. Ask for higher coverage and you will reliably get higher coverage. You will not reliably get more verification.

Coverage still has one legitimate use: finding code no test ever touches. Treat it as a map of blank space, never as evidence that the filled-in space works.

The measurement that works: mutation testing

Mutation testing asks the question you care about. It deliberately breaks your source code, one small change at a time, and reruns the tests against each broken version. If a test fails, the mutant is "killed" and the suite noticed. If every test still passes, the mutant "survived" and you have proof that this breakage would ship unnoticed.

The mutations are small and mechanical, which is what makes them a fair test of assertion quality:

• >= becomes >, so an off-by-one at a boundary
• && becomes ||
• a returned value becomes null, 0, or ""
• a conditional block is removed entirely
• a method call is stripped and its return replaced with a default

Your mutation score is killed mutants divided by the mutants that could have been killed. A suite with 90% line coverage and a 30% mutation score is executing a lot of code and checking almost none of it. That combination is common in generated suites, and until you run the tool you cannot see it.

How to run it

The tooling is mature and predates this argument by fifteen years. JavaScript and TypeScript: Stryker.

npm i -D @stryker-mutator/core
npx stryker init
npx stryker run

Scope it in stryker.config.json first, because mutating everything is slow:

{
  "mutate": ["src/pricing/**/*.ts", "src/auth/**/*.ts"],
  "testRunner": "vitest",
  "thresholds": { "high": 80, "low": 60, "break": 50 }
}

Python: mutmut run, then mutmut results and mutmut show <id> to read each survivor. Java: mvn org.pitest:pitest-maven:mutationCoverage. Go: go-mutesting ./....

Three practical notes decide whether this survives contact with your CI.

Cost. Mutation testing runs your suite once per mutant, so a naive full-repo run can take hours. Scope it to the modules where a bug is expensive: pricing, permissions, billing, anything touching money or access. Most runners also support mutating only changed files, which turns it into a per-pull-request check that finishes in minutes.

Cadence. Full run nightly or weekly, diff-scoped run on pull requests. Nobody needs a mutation score on every commit.

Equivalent mutants. Some mutations produce code that behaves identically to the original, so no test could kill them. They are the noise floor. Do not chase 100%, and do not let a survivor list get dismissed wholesale because a few entries are unkillable.

The survivors are the report

The score is a headline. The value is the list of what survived.

A surviving mutant is a specific, reproducible sentence: "we changed this line to return null and every one of your 340 tests still passed." That is not an opinion about AI-generated test quality. It is a defect report about your suite, with a file and a line number.

Read the survivors in order of how much you would hate the corresponding real bug. Most teams find the same shape: tests cluster around happy paths, and survivors cluster at boundaries, error branches, and the code that decides whether someone is allowed to do something.

What a good assertion looks like

Mutation testing tells you where the hollow assertions are. Fixing them means knowing what a real one looks like. Here is the pattern that fails silently:

test('summarises the invoice', () => {
  const summary = summariseInvoice(invoice)
  expect(summary).toBeDefined()
  expect(typeof summary).toBe('string')
})

That passes when summariseInvoice returns "". It passes when it returns the wrong customer's name. It passes after you replace the function body with return "", and it sits in a coverage report looking like coverage.

Here is the tautology, which is worse because it can never fail:

test('formats the total', () => {
  expect(formatTotal(cart)).toBe(formatTotal(cart))
})

And the mock-level version, expect(paymentClient.charge).toHaveBeenCalled(), which checks that your code called your mock. The charge can fire correctly while the response is dropped and the user sees a blank screen.

The versions that catch bugs share three properties: a concrete expected value, chosen independently of the implementation, describing something a user would notice.

test('applies SUMMER to a two-item cart', () => {
  const cart = { items: [{ price: 2500 }, { price: 2000 }], code: 'SUMMER' }
  expect(formatTotal(cart)).toBe('$40.50')
})

test('rejects an expired code and leaves the total unchanged', () => {
  const cart = { items: [{ price: 2500 }], code: 'WINTER2024' }
  expect(() => applyCode(cart)).toThrow('code expired')
  expect(formatTotal(cart)).toBe('$25.00')
})

'$40.50' is a number a person worked out from what the discount is supposed to do. That independence is the whole thing: an expected value copied from running the code is a screenshot of current behaviour, not a statement about correct behaviour.

The failure mode where the agent edits the test

The sharpest complaint deserves the sharpest answer.

Give an agent a red suite and the instruction "make the tests pass", and it has two paths. One is diagnosing a bug in unfamiliar application code and making a real fix. The other is widening a matcher, replacing an exact value with a contains, raising a timeout, or marking the case skipped. The second is cheaper and reaches the stated goal, so a system that only knows how to want green will find green.

A better model does not fix this, and neither does "do not weaken assertions" in a prompt. What fixes it is making a second outcome expressible: the repair path has to be able to conclude the application is wrong and report a bug with the test left failing. A pipeline whose only representable end state is a passing suite will produce one, and destroy the information you needed on the way.

Two mechanics make that checkable. Repairs arrive as a reviewable diff, because "the test was updated" and "the assertion was deleted" look identical from a green checkmark. And weakening is worth its own number: assertions loosened, cases skipped, timeouts raised, per week. If that climbs while the suite stays green, your suite is being quietly retired. The broader review structure is covered in where the human belongs when agents write the tests.

The end-to-end version: break the app on purpose

Mutation tools operate on source code, so they are a unit and integration-level instrument. End-to-end suites need the same idea run by hand, and it is worth an hour a quarter.

The drill: pick a real defect your team shipped in the last six months, revert the fix on a branch, run the end-to-end suite, and see whether it goes red. Do that for five past bugs. The result is your suite's catch rate against the kind of bug your product actually produces, which beats any coverage figure.

Cheaper variants: feature-flag off a capability the suite covers, change a validation rule so the app accepts input it should reject, break a permission check so a lower-privilege user reaches a restricted page, or return an empty list from one API handler.

That last one most often exposes the assertion-free walkthrough. A test that opens a dashboard and asserts the page loaded will pass happily against a dashboard with no data in it.

Where the test came from changes its properties

There is a real, structural difference between two things that both get called AI-generated tests.

A test generated by reading source code has one source of truth about correct behaviour: the implementation whose correctness is in question. Asked to write a test for a function, the most honest thing a model can produce from that input is a description of what the function does. That is the recipe for a tautology, and it explains why the failure mode is so consistent rather than random.

A test derived from an observed run in a real browser has a different source of truth. Something walked the application, submitted the form, and recorded what rendered: the confirmation text, the displayed total, the value that persisted after a reload, the error shown on bad input. Those assertions describe the running system rather than restating the code, which is what an oracle is.

Be straight about the limit. An observed run encodes observed behaviour, so if the behaviour was already wrong when it was observed, the test now defends the bug. That makes the first review of a new suite the most valuable review it will ever get, because that is the one where you decide what the application should do rather than check a change to it.

The honest conclusion

AI-generated tests can absolutely be worthless. A suite of them can report high coverage, green builds, and confident test names while verifying nothing, and no amount of model improvement removes that possibility, because the failure is in the objective rather than the writing.

What separates a suite that catches bugs from one that does not is process, and the process is checkable:

1. Where did the assertions come from, an observed run or a reading of the source?
2. Can the system conclude the application is broken, or is green its only expressible outcome?
3. Do repairs arrive as diffs a person reviews and merges?
4. What is the mutation score on the code where a bug would be expensive?
5. When you revert five real past bugs, how many does the suite catch?

Run question five first. It is the least sophisticated measurement here and it settles the argument fastest.

How Shiplight is arranged for this

Tests are derived from a real session: the agent walks your application in an actual browser and writes what it observed, so assertions come from the running system rather than from a reading of the code. The output is readable YAML in your own git repository, short enough that the person who knows what the feature should do can read it like a spec and say the expected value is wrong, which catches a bad assertion before mutation testing has to. When a test fails, /shiplight fix reproduces it, diagnoses the root cause, and repairs the test when the interface changed. When the application itself is broken, it reports the bug instead of editing the test to pass. In CI it will diagnose, fix, rerun and open a pull request, and it never merges on its own.

Two limits, since a page about honest measurement should not end on an unqualified claim. Authoring runs through your coding agent's own subscription and needs no Shiplight account, but executing a test file needs an LLM key, ours or your own. And Shiplight covers web applications, so native mobile and desktop estates are outside its scope.

FAQ