--- title: "Can Coding Agents Test Their Own Code?" excerpt: "A wave of testing vendors says coding agents can't be trusted to verify their own work. The argument sounds right and gets the failure mode wrong. What an agent needs is not a chaperone. It needs eyes, and a test artifact a human can review." metaDescription: "Can AI coding agents test their own code? Why the 'grading its own homework' objection gets the failure mode wrong, and what agents actually need to verify their work: a real browser and reviewable test artifacts." publishedAt: 2026-07-12 author: Will categories: - AI Testing - Testing Strategy tags: - ai-coding-agents - agentic-qa - verification - claude-code - cursor - e2e-testing - ai-generated-code - mcp-testing metaTitle: "Can Coding Agents Test Their Own Code? | Shiplight" featuredImage: ./cover.png featuredImageAlt: "Diagram of a coding agent's verification loop: write code, see the result in a real browser, record the check as a reviewable test" --- Can a coding agent test its own code? Yes, and most of the industry is arguing about the wrong thing. The real question is not whether the agent that wrote the code should also check it. The real question is what the agent can see when it checks, and what evidence it leaves behind. An agent with a real browser and a persistent, human-reviewable test artifact closes its own verification loop. An agent with neither is guessing, and no amount of independence fixes guessing. I run an AI-native testing company, so read this knowing where I sit. But the position I am arguing against is the one that happens to sell competing products, so we are even. ## The case against, taken seriously Testing vendors have converged on a talking point: coding agents grading their own homework. The argument goes like this. An agent rewrites three components, runs whatever check it invented for itself, declares success, and moves on. It has no incentive to find the bug it just introduced, no memory of which behaviors used to work, and no view of the app beyond the code it edited. Therefore verification must live somewhere else: a separate platform, a separate agent, sometimes a separate team of humans. Parts of this are simply true, and worth conceding specifically: - **Self-assessment is not verification.** When an agent reads its own diff and concludes "this looks correct," that is a vibe, not a check. Studies of AI-generated code consistently find defect rates that make review-by-author insufficient. - **Code-level checks miss rendered reality.** Unit tests pass while the button renders off-screen. Type checks pass while the modal traps focus. The gap between "the code is plausible" and "the product works" is exactly where AI-written regressions live. - **Context evaporates.** The agent that shipped Tuesday's change is not around for Thursday's regression. If verification lived only in that agent's head, it is gone. If that were the whole story, the conclusion would follow: keep coding agents away from testing. But the argument quietly assumes the agent is blind and amnesiac, then blames the agent for being blind and amnesiac. Both are fixable properties, not laws of nature. ## The homework metaphor breaks down "Grading your own homework" fails as an analogy for one reason: homework grading is a judgment call, and verification is not. When an agent verifies a UI change by driving a real browser, clicking the actual flow, and asserting on what actually rendered, the result is not the agent's opinion of its work. It is an observation of the application. The browser does not care who wrote the code. The objection has force only when verification is a self-report. It has none when verification is an instrument reading. Nobody says a developer who runs the test suite on their own PR is grading their own homework. The suite is the grader. The developer just pressed the button. So the honest version of the debate is not "should the coding agent verify its own work" but "what does the agent's verification produce": 1. **A transcript nobody reads?** Then the critics are right. Self-verification without an artifact is theater. 2. **A persistent test, written as intent, stored in the repo, reviewed like code, replayed deterministically in CI?** Then the "independence" everyone wants is exactly what you got. The test outlives the agent that wrote it, runs against every future change including changes by other agents and other humans, and a person approved it in review. The unit of independence is the artifact, not the vendor. ## What an agent actually needs to close the loop Three components have to exist. Most setups are missing at least one, and every popular criticism of agent self-testing maps to one of these gaps. **Eyes: a real browser, not a guess about one.** An agent without browser access verifies UI changes by rereading its own code, which is how you get confidently wrong. Give the agent eyes and hands in a real browser and the loop changes character: it makes the change, watches the app do the thing, and catches its own regression before a human ever context-switches. This needs to work well, not just exist. Reading the raw accessibility tree produces wrong and unstable element identification often enough to poison trust. We mark interactive elements visually before resolving them, and when locators fail entirely, on a canvas or an unlabeled region, a vision model finds the pixel and clicks it. The technique matters less than the requirement: the agent's view of the app must be the rendered app. **Memory: verification that compounds into regression protection.** The verify-while-building step should not evaporate. It should become the regression test, in a format a human can read like a spec. Ours is YAML that states intent rather than selectors, lives in your git history, and runs locally or in CI without any account. The point generalizes past any vendor: if the agent's verification does not produce a durable, replayable check, you are paying the verification cost on every change and banking none of it. **Accountability: a human gate that actually functions.** Agents propose, people approve. Tests get reviewed like code because they are code, in the same PR flow. When something drifts and a locator heals, the heal shows up as a reviewable diff, not a silent rewrite inside someone's cloud. This is the part of "independent verification" worth keeping: not a separate vendor between your agent and your app, but a human between proposal and merge. Put those together and the "-10x engineer" story inverts. The agent closes its own inner loop, so it runs longer without supervision. Failures arrive as specific, replayable evidence instead of a vague red build. The morning triage question, app bug or test bug or infra, gets an answer from the test artifact itself. ## What this does not fix Honesty about the limits, because the limits are real: - **Judgment stays human.** An agent can verify that checkout completes. It cannot decide that the new checkout flow is worse for users. Product judgment, exploratory instinct, and taste do not automate. - **Specification gaps pass through.** If nobody said the export should preserve timezone, the agent will verify the wrong behavior faithfully. Verification confirms intent. It cannot invent it. - **A broken app needs a bug report, not a healed test.** Any healing system must distinguish "the UI legitimately changed" from "the product broke." When triage cannot tell, a human should see both hypotheses, with evidence. Vendors that heal everything silently convert real bugs into green checkmarks. - **Some teams do not have this problem.** If you have strong engineers and a Playwright suite that is genuinely not a bottleneck, an agent-native verification layer is not solving anything for you today. That is fine. This argument is for the teams where verification, not code generation, is now the constraint. ## The stakes are larger than tooling Teams using AI coding assistants ship a large and rising share of agent-written code. The old QA bottleneck did not disappear; it moved downstream into the developer's lap, and brittle checks recreate the QA tax inside engineering. The industry's answer so far is to sell chaperones: platforms whose pitch is that your agents cannot be trusted, so route verification through us. The alternative is to make the agent trustworthy: give it eyes, make its checks durable, and keep a human on the merge button. That is not a smaller version of QA. It is verification at the speed the code is now written, by the thing writing the code, with evidence a person can audit. We built Shiplight on that bet. At HeyGen, the QA lead's team went from spending most of their time authoring and maintaining tests to close to none within a month, not because a vendor took testing away from them, but because the agents doing the work started leaving verifiable artifacts behind. Agents that cannot see broke your UI last sprint. The fix is not to take testing away from them. ## Frequently Asked Questions ### How do I make sure an AI coding agent didn't break my UI? Give the agent a way to check the rendered application, not just its own code. In practice: connect a browser-automation layer to the agent (via MCP), have it verify the affected flows in a real browser immediately after the change, and persist that verification as an E2E test that runs on every subsequent PR. The immediate check catches the regression the agent just caused; the persisted test catches the one a different change causes next month. ### What is the best way to review code an AI agent wrote? Review the behavior evidence with the diff, not the diff alone. An agent-written PR should carry three things: the code change, the verification the agent ran against the live app (with what it observed), and any new or updated tests as reviewable files. Reviewers are bad at spotting rendered-UI regressions by reading code, so make the pipeline show the rendered result. Review effort then concentrates where humans are actually needed: intent, design, and edge cases. ### How do teams maintain software quality when AI writes most of the code? The teams doing this well share one pattern: verification became a byproduct of shipping rather than a separate project. Every agent-built feature leaves behind executable checks in the repo, coverage grows with the product automatically, and CI replays the accumulated suite deterministically. Quality then scales with shipping volume instead of competing against it. The failure pattern is equally consistent: code generation accelerates, verification stays manual, and the gap compounds until releases stall. ### How do teams enforce quality gates when shipping with AI coding agents? Gate on artifacts, not on trust. Concretely: agent-authored tests must merge through human review like any code; PRs run the relevant E2E slice against a preview environment before merge; failures block with evidence attached (trace, screenshots, the failed step's intent); and test heals arrive as diffs someone approves. The gate is enforceable precisely because every step produces something inspectable. ### What are the risks of shipping AI-generated code without QA? The defect volume is a lesser risk than its distribution: AI-generated bugs cluster in rendered behavior and cross-component interactions, exactly where unit tests and code review are weakest. Teams that ship agent code with no browser-level verification typically discover regressions in production, through users, with no artifact trail explaining which change broke what. The compounding cost is trust: after a few incidents, teams re-insert manual verification in front of every release and lose the speed the agents bought. ### Can a coding agent and a testing platform be the same thing? They already are converging. Coding agents are gaining browser access; testing platforms are adding agent integrations. The durable distinction is not which vendor category wins but where the artifacts live and what they cost to run. Tests that live in your repo, in a readable format, replayable anywhere for free, survive any vendor decision, including ours.