What Is Agent-Native Testing? The Category, the Requirements, and How to Tell

Agent-native testing is end-to-end testing where an AI coding agent is the operator: it walks the application in a real browser, writes the tests, runs them, diagnoses failures and repairs them, while the human reviews the result and owns the decision to ship.

The distinction it draws is not about how much AI is inside the tool. Almost every testing product has AI in it now. The distinction is about who is holding the work. In agent-native testing the agent holds it end to end. In everything else, a person does, and the AI makes their steps faster.

This page defines the category, states what it actually requires, and gives you a way to check a claim rather than take it.

The one-sentence test

Point at a testing tool and ask: can an agent do the whole job, or does it hand work back to a human halfway through?

Most tools hand it back. An agent can trigger a run but cannot read the result. It can generate a test but cannot save it anywhere the agent can find again. It can see a failure but has no way to tell an application bug from a stale selector. Each handback is a person, and the people are the reason test suites decay.

Why the category exists

End-to-end testing has always had an unusual cost structure. Running tests is cheap. Writing and maintaining them is not, and the maintenance cost is the one that compounds.

The mechanism is well understood. Tests bind to the interface through selectors. The interface changes constantly. Every change breaks some tests, and someone has to work out which breaks are real regressions and which are the suite complaining about a renamed class. Teams solve this with people: their own QA engineers, or an outsourced service, or senior engineers who lose a morning a week to it.

That cost did not go away when coding agents arrived. It got worse, because the rate of interface change went up. A team shipping three UI changes a week and a team shipping thirty have the same test suite and very different maintenance bills.

The premise of agent-native testing is that the work that was being done by people is work an agent can do, provided the tooling is built so an agent can reach it. That proviso is the whole category.

The four requirements

These are architectural, and each one is checkable. A tool that misses any of them cannot support an agent operator regardless of what its marketing says.

1. The actions are callable, not clickable

The agent has to be able to invoke the tool's capabilities. In practice that means a protocol such as the Model Context Protocol, a CLI, or agent skills, or some combination. If the only way to start a run or open a browser is to click it in a dashboard, the agent is locked out of the product entirely.

This is the requirement most vendors can satisfy quickly, which is why it is also the one most often pointed at as proof. It is necessary and it is nowhere near sufficient.

2. The tests are files

The output of authoring has to be an artefact the agent can read back, diff and edit. A test stored as rows in the vendor's database fails this: the agent cannot open it, cannot see what changed, and cannot fix it. A test stored as a file in your repository passes: the agent edits it the way it edits any other file, and your existing code review sees the change.

This requirement has a side effect worth naming. Once tests are files, they branch with your code, they show up in pull requests, and leaving the vendor is possible. That is not an accident of the design, it is what the design implies.

3. The result is readable

The agent has to be able to check what happened. Triggering an action is half a loop; closing it requires an outcome the agent can parse. This is the requirement most often missing in practice. A tool will happily let an agent start a test run and then report the outcome only as a rendered dashboard, which means the agent is flying blind and a human has to come back and interpret.

For end-to-end testing specifically, "readable" has to include the evidence: screenshots, traces, console and network output. A failure message alone rarely tells you whether the application broke or the test did.

4. The human gate is explicit and late

Every serious definition of agent-native includes human oversight, so the question is never whether a person is involved. It is where. Agent-native testing puts the human at review and approval: reading the test like a spec, deciding whether the change is right, owning the merge. It does not put the human in the middle of the loop clicking next.

A system that asks for approval at every step is not agent-native, it is a human tool with an agent typing. A system with no human gate at all is not agent-native either, it is unreviewed automation, and it is the thing engineers are right to distrust.

What it is not

Three adjacent things get called agent-native testing and are not.

AI-augmented testing. A console with AI features: smart selectors, suggested assertions, a natural-language recorder. Genuinely useful, and the human still drives every step. The AI accelerates the person; it does not replace their position in the loop.

Agentic testing. "Agentic" describes autonomous behaviour, meaning a system that plans and acts. A product can be agentic inside a closed console: its agent plans and acts, but only within the vendor's runtime, and your agent cannot reach it. Agentic is a claim about behaviour, agent-native is a claim about architecture. See what is agentic QA testing for that distinction in full.

Managed QA. A service where the vendor's engineers, increasingly assisted by AI tooling, write and maintain your tests. The suite stays green and the mechanism is staffing. The tests live on the vendor's infrastructure and the knowledge your product accumulates sits outside your engineering organisation. Whatever the marketing says, the operator is a person; they are just not your person.

None of these are illegitimate products. They are different products, and the reason to separate them is that they fail in different ways and cost different amounts over three years.

What actually changes

Three things, and they are worth being concrete about because the category claim is otherwise abstract.

Coverage becomes a byproduct rather than a project. If verifying a change and writing the covering test are the same action performed by the same agent, coverage grows as you ship rather than in a separate initiative that gets deprioritised. The suite is a side effect of work you were doing anyway.

Maintenance stops being headcount. When the UI changes, the agent re-derives the step from the stated intent rather than failing on a stale selector, and the repair arrives as a reviewable diff rather than a ticket. Teams that had a person on suite upkeep get that person back.

Failure triage shortens. The expensive part of a red build was never the fix, it was the ambiguity: app bug, test issue, infrastructure, or config. An agent that can read the run evidence and reproduce the failure can classify it and propose the right action, which is the difference between a notification and a diagnosis.

How to check a claim

Five questions, in the order that finds the truth fastest.

1. Can my agent call it? Ask for the MCP server, the CLI, or the skill. If the answer is a dashboard walkthrough, that is the answer.
2. Where does the test live? If the artefact is in their cloud, your agent cannot maintain it and neither can you without them.
3. Can the agent read the result? Including the evidence, not just pass or fail.
4. What happens when it fails? A diagnosis and a proposed fix, or a red mark for a person to interpret.
5. Where is the human? Supervising at review, or in the middle approving each step.

Gartner named the failure mode here "agent washing": products described as agentic or agent-native without the underlying capability, and its research line estimated that only a small fraction of vendors making the claim were genuinely building the thing. Treat the label as worth nothing and the five answers as worth everything.

Where Shiplight sits

Shiplight is the agent-native testing platform, and it is worth stating that against the four requirements rather than as an adjective.

Callable: it installs into your coding agent as a browser MCP server plus skills, so Claude Code, Cursor, Codex and others invoke it directly. The MCP server is deterministic and needs no model key of its own; your agent supplies the reasoning.

Files: tests are readable YAML in your own git repository. They branch with your code and show up in pull requests. The YAML transpiles to Playwright at run time, so there is a real exit path rather than a one-way door.

Readable results: runs return structured output plus the evidence, screenshots and traces included, and the agent can read results back rather than waiting for a human to open a dashboard.

Human gate: you review the test like a spec and own the merge. In CI, Shiplight will diagnose a failure, apply the fix, rerun and open a pull request, and it never auto-merges. If the application itself is broken, it reports the bug instead of editing the test to make it pass.

Two limits worth stating plainly. Authoring runs through your coding agent's own subscription and needs no Shiplight account, but executing a test file needs an LLM key, ours or your own. And Shiplight is built for web applications, so native mobile and desktop estates are outside its scope.

FAQ