Test generation has been the most oversold capability in testing for about two years. Almost every vendor claims it. Almost none of it survives contact with a real application, because generating a plausible test is easy and generating one that passes against a running app is not.
So it is worth paying attention when the project that most teams actually use for browser automation ships its own version. Playwright now includes three first-party test agents, and the design is more careful than the category norm. This is a look at what each one does, what it is good for, and where the approach stops.
Installing them
One command writes the agent definitions into your repository:
npx playwright init-agents --loop=claudeThe --loop value names the environment the agents will run inside. The documented options are vscode, claude, codex and opencode, which covers VS Code with Copilot, Claude Code, OpenAI's Codex and OpenCode. VS Code needs version 1.105 or later.
The definitions are generated files, and the docs are explicit that you should regenerate them whenever you update Playwright, so the agents pick up new tools and instructions. That is worth putting in your upgrade checklist, because a stale agent definition will keep working while quietly using an older tool surface.
The planner
The planner explores your application and produces a markdown test plan covering one or many scenarios.
It takes three inputs. A clear request, which is the part most people underestimate. An optional product requirements document, if you have one worth pointing at. And a seed test, which is the input that makes the whole thing work.
The seed test is a small file at tests/seed.spec.ts that does nothing interesting on its own. Its job is to be run by the planner so that everything your test environment needs actually happens: global setup, project dependencies, fixtures and hooks. Without it, the planner is exploring your app in whatever state a cold browser lands in, which for most real applications is a login page. With it, the planner explores as an authenticated user inside your real fixture setup.
The output is a file like specs/basic-operations.md, described as human-readable but precise enough to generate from. That intermediate artefact is the best design decision in the set. A markdown plan is something a QA lead or a product manager can read and correct, and correcting a plan costs a fraction of correcting generated code. It also means the expensive exploration happens once and can be re-generated from without re-walking the app.
The generator
The generator takes the markdown plan and produces executable Playwright test files under tests/.
The important detail is that it verifies selectors and assertions live as it performs the scenarios. It is not writing code from a description of your app and hoping. It drives the browser through each scenario as it writes, so the locators it emits are ones it just used successfully.
The docs are honest about the result: generated tests may include initial errors, which the healer can then repair. That is a fair description of where automated generation currently sits, and stating it beats the alternative claim.
The healer
The healer is given a failing test name and runs a repair loop. It replays the failing steps, inspects the current UI to find equivalent elements or flows, suggests a patch such as a locator update, a wait adjustment or a data fix, and re-runs until the test passes or guardrails stop the loop.
Two things about the output deserve attention. First, it is a patch to your test file, a normal diff you review and commit. Second, the documented outcome is a passing test, or a skipped test if the healer believes the functionality is genuinely broken. That second branch matters more than it looks: it is the difference between a repair tool and a tool that makes red things green. A healer that always succeeds is a healer that will eventually rewrite a test around a real bug.
How it fits a suite you already have
The fit is good, and that is the main reason to look at it.
The output is ordinary Playwright test files in your repository. They run under your existing config, in your existing CI, alongside tests written any other way. There is no runtime to adopt and no account to create. If you decide the agents are not for you after a week, you delete the agent definitions and keep the tests.
The realistic entry point is coverage you have been meaning to add and have not: a flow that everyone agrees should be tested and that nobody has had a free afternoon for. Point the planner at it, read the markdown plan, correct it, then generate. Reviewing a plan before any code exists is a genuinely better loop than reviewing a pull request full of generated test code and trying to reconstruct what it was supposed to check.
Two practical notes. Get the seed test right before anything else, since a planner exploring an unauthenticated app produces a plan about your login page. And treat the plans in specs/ as artefacts you keep and edit, not as scratch output, because that is what pays off on the second and third pass.
What it solves, and what it does not
It solves the authoring bottleneck, which is real. Writing the first version of an end-to-end test is slow work with a low ceiling on how interesting it can be, and having an agent walk the app and draft it while checking its own locators is a genuine improvement over a human doing the same thing more slowly.
It also solves a slice of maintenance: the repair sits in a loop rather than in a person's morning, and the patch arrives as a diff.
What it does not change is what the test is. The artefact is Playwright code bound to locators, and those locators were correct at the moment they were generated. When the UI changes, the test fails, and something has to run to fix it. The healer makes that fix cheaper. It does not make the binding disappear.
That has three consequences worth being concrete about.
Repair happens after a failure, at development time. The sequence is: change ships, test goes red, someone runs the healer, a patch appears, a human reviews and merges. That loop is shorter than the manual one, and it is still a loop with a red build and a person in it.
A patched locator is a new binding, not a removed one. The healer updates the selector to the one that works today. The next redesign breaks the new one the same way.
In CI, a healer is a development-time tool. You are not going to run an unattended agent that rewrites test code inside your pipeline, and you should not want to. So the repair work still lands on a branch, in review, with the delay that implies.
None of this is a criticism of the design. It is what an agent that produces code files can do. The limits belong to the artefact, not the agents.
Generate-time versus run-time resolution
There are two mechanisms for keeping an end-to-end test working while the UI changes, and the difference is when the element gets resolved.
Generate-time resolution. The intent, "click the Save button", is turned into a locator once, at authoring time, and that locator is what ships in the file. The intent is now gone: the file records the how, not the what. When the locator stops matching, the test fails, and a repair step has to recover the intent by inference from the failing code. Playwright's healer does that recovery well, and it is still recovery of information the file no longer contains.
Run-time resolution. The intent stays in the test file as the durable thing, and the element is resolved when the test runs. A stable UI resolves from a cached locator with no model call and no speed penalty. A changed UI misses the cache, the step re-resolves from the stated intent, and the run continues rather than going red. The repair is not a code change, so there is no diff, no review and no branch for the class of change where the user-visible behaviour did not move at all.
The honest trade-off runs both ways. Generate-time output is plain code: fully inspectable, debuggable with tools your team already knows, and free of any runtime dependency. Run-time resolution costs a model call on a cache miss and asks you to trust a resolution step you did not write, in exchange for a class of failure that stops happening.
The question that decides it is what fraction of your red builds are real regressions. A team shipping UI changes weekly can absorb a repair loop. A team where an agent rewrites three components before lunch is looking at a different arithmetic, because every one of those changes is a chance for the suite to go red about something that is not broken.
Where Shiplight sits
Shiplight is built on the run-time side of that split, and the mechanism is the whole difference.
Tests are readable YAML in your own repository, written from intent rather than selectors. At run time each test transpiles to Playwright plus Shiplight's AI SDK, which is why every Playwright browser and feature is available. Deterministic steps cache their resolved locator and run at full Playwright speed with no model call. On a cache miss the step re-resolves from the original intent and the run continues, and the hosted cache updates immediately, so the next run already has the fix with nothing committed. AI steps and assertions are not cached, by design: they call the model every run, which is what lets them handle what a fixed selector cannot.
If you already have a Playwright suite, keep it. The transpiled output runs alongside it under the same config, so both kinds of test run together, and the transpilation is also the exit path if you ever want one. If you do not have a suite today, you do not need to build one first.
Two limits, stated plainly. Authoring runs through your coding agent's own subscription and needs no Shiplight account, but executing a test file needs an LLM key, ours or your own. And Shiplight covers web applications, so native mobile and desktop estates are outside its scope.
Should you adopt them
Yes, if you have a Playwright suite you intend to keep and an authoring backlog. The install is one command, the output is code you own, and the cost of finding out is an afternoon. The planner and the markdown plan alone are worth the trial, independently of whether you keep the healer in your loop.
Be more careful if your problem is maintenance rather than authoring. Faster repair of selector-bound tests is a real improvement over slower repair, and it is still the same category of work. Measure it before you decide: count what share of your red builds over the last month were real regressions. If most of them were not, the thing to change is when your tests resolve, not how quickly you patch them.
FAQ
What are Playwright test agents?
Three first-party agents that ship with Playwright: a planner that explores your app and writes a markdown test plan, a generator that turns that plan into Playwright test files while verifying selectors live, and a healer that repairs failing tests. You install their definitions with npx playwright init-agents.
How do I install them?
Run npx playwright init-agents --loop=<environment>, where the environment is vscode, claude, codex or opencode. VS Code needs version 1.105 or later. Regenerate the definitions whenever you upgrade Playwright, since the agent definitions are generated files that pick up new tools and instructions.
What is the seed test for?
tests/seed.spec.ts is run by the planner so that your global setup, project dependencies, fixtures and hooks all execute before exploration begins. It is what lets the planner explore your app as an authenticated user in a realistic state rather than staring at a login page. Getting it right is the highest-value setup step.
Does the healer fix tests in CI automatically?
It is a development-time loop rather than an unattended pipeline step: it replays the failing steps, inspects the UI, proposes a patch, and re-runs until the test passes or guardrails stop it. The output is a diff you review, or a skipped test if it concludes the functionality is genuinely broken. That second outcome is what stops it from rewriting a test around a real bug.
How is this different from tests that resolve intent at run time?
The healer repairs a locator after a test has failed, and the fix is a code change you review and merge. Run-time resolution keeps the intent in the test file and resolves the element during the run, so a UI change that did not alter behaviour re-resolves and the run continues without a red build or a diff. See intent-based testing for the model.
Do the agents replace a QA engineer?
No, and the design does not claim to. The markdown plan exists precisely so a person reads and corrects the scenarios before any code is generated, which is judgement work about what should be tested. What moves off a person's plate is drafting and repair, not deciding what coverage is worth having.
