Shiplight vs Playwright: Which Should You Use? (2026)

Shiplight and Playwright approach end-to-end testing in two different ways. With Playwright you write and maintain the tests yourself: code, selectors, and the upkeep every time the UI changes.

With Shiplight you describe what to test in plain language, and coding agents author the tests, keep them green, and self-heal them when the UI shifts. You do not need to know or write Playwright to use Shiplight.

Under the hood Shiplight runs on the Playwright engine, the same Chromium, Firefox, and WebKit execution at the same speed, so there is no reliability trade-off. And if you already have a Playwright suite, Shiplight is compatible and runs alongside it.

The decision you are actually making

Most teams weighing this are deciding how to do E2E testing at all: own and maintain browser-automation code, or let a platform author and self-heal it. That is the real comparison.

It is not two engines, since underneath it is the same Playwright either way, but two working models: code you own and patch, versus intent an agent authors and heals.

We build Shiplight, so we have a stake in this comparison. The honest version: Playwright is the best open-source browser automation framework available, which is exactly why we built on it instead of against it. This page covers how each approach works, what Shiplight does differently, and the cases where raw Playwright is the right call.

Quick Comparison

How Playwright Works: Code and Selectors

Playwright is Microsoft's open-source browser automation framework. It drives Chromium, Firefox, and WebKit, supports JavaScript/TypeScript, Python, Java, and .NET, and ships first-class tooling: auto-waiting, codegen, a trace viewer, network interception, fixtures. You write tests as code and run them in your own CI.

A Playwright test looks like this:

import { test, expect } from '@playwright/test';

test('user can create a project', async ({ page }) => {
  await page.goto('/dashboard');
  await page.getByRole('button', { name: 'New Project' }).click();
  await page.getByLabel('Project name').fill('My Project');
  await page.getByRole('button', { name: 'Save' }).click();
  await expect(page.getByText('My Project')).toBeVisible();
});

Playwright's genuine strength is scoped and real: full programmatic control over the browser, in a mature, free, multi-language framework. For an engineer who wants to own every interaction and assertion in code, nothing on the market beats it, which is why Shiplight runs on it rather than around it.

The cost lives in that same design: every locator in that test is a maintenance liability. When the UI changes, selectors break, and the code they live in has to be patched. Teams with large suites report the pattern consistently: authoring slows down and maintenance grows until it dominates.

One Head of QA we work with measured roughly 60% of their time going to authoring and maintaining Playwright tests before switching to intent-based, self-healing tests brought it to near zero within a month.

How Shiplight Works: Intent and Self-Healing

With Shiplight you never write Playwright. Tests are YAML files stating user intent, they live in your git repo next to the code they verify, and the Shiplight MCP server and Skills install into the coding agent, so the agent that builds a feature verifies it in a real browser and writes the regression test in the same session.

The YAML format exists for the humans: the agent authors and maintains the suite, and readable intent is what lets your team review its work like a spec and stay in the loop.

Custom logic stays available too: steps drop into inline JavaScript, and complex flows can be fine-tuned in the local debugger with screenshots, traces, and step-through.

The same test in Shiplight:

goal: Verify user can create a new project
statements:
 - intent: Navigate to the dashboard
 - intent: Click "New Project" in the sidebar
 - intent: Enter "My Project" in the project name field
 - intent: Click the Save button
 - VERIFY: the project appears in the project list

Five mechanisms do the work:

Set-of-marks visual prompting

Before resolving a step, Shiplight marks the interactive elements on the rendered page, then resolves locators from what is actually visible. Playwright's role-based locators come from the accessibility tree, which works well when that tree is well-formed and thins out when it is not.

Resolving from the rendered page holds up on pages that break accessibility-tree locators, and per-build dynamic IDs never enter the test: intent re-resolves against the rendered page, so churning data-id attributes are a non-event.

AI-judged assertions

A VERIFY: step states an outcome in language, and the model judges the rendered page against it. Playwright's expect() compares against exact strings and selectors supplied in advance. The clearest example is localization: "VERIFY: the page is fully localized in Japanese, with no untranslated strings" is one Shiplight step.

The same check cannot be expressed as a string assertion; per-locale expected values only prove the strings you thought to hardcode, not that the page reads correctly.

Vision-model fallback

When no locator can reach an element at all (canvas, drawing surfaces, custom-rendered regions), Shiplight falls back to a vision model that finds the pixel and clicks it. Raw Playwright has no equivalent, so those flows usually go untested.

Step-level locator cache with online healing

Resolved locators are cached and committed to your repo, so stable tests run at plain-Playwright speed. When the UI changes, the broken locator heals at run time from the test's stated intent, and larger changes arrive as a PR diff from the triage agent for a human to review. Heals are visible changes in git history, never silent rewrites.

Batch authoring through your agent

The agent writes test files directly rather than round-tripping each action through a server, which is why regression suites get built in days, not months: first suites of roughly 300 tests within the first week.

The trade-offs, stated just as plainly

Shiplight is web only, with no native mobile or desktop testing. The hosted platform adds a vendor relationship. And it is years younger than Playwright, with a smaller community.

The Real Difference: Tests That Are Easier to Live With

Execution is not the differentiator, because it is the same engine at comparable speed (cached locators run at selector speed). The difference is the whole life of a test: writing it, reading it, and keeping it working, over months, at the same headcount.

Easy to write, even by a coding agent

There are no selectors to hand-craft and no framework API to learn. A step is a sentence of intent, so an agent can author a full suite by walking your app, and a human can too.

Easy to read and review

YAML intent reads like a spec, which is what keeps a person in the loop instead of rubber-stamping code they did not write. Self-heals and larger fixes arrive as reviewable git diffs, so the human is still the final check on every change. Nothing happens silently.

Non-flaky and low-maintenance

Selector churn and brittle timing are the usual sources of flaky E2E suites. Intent re-resolution, the action cache, vision fallback, and online healing take those failure modes off the table, so a UI change is absorbed instead of turning into a red build and a morning of triage.

To be fair to Playwright: you can bolt agents onto it too, and its own Test Agents can suggest fixes, and in the end a human reviews the result either way. The difference is that Shiplight's AI capabilities, auto-heal, action cache, intent authoring, and vision, are built in and work together.

The result is tests that are cheaper to write, less flaky, easier to review, and less work to live with over time.

That is the same team's hours going further: fewer failures and less flakiness, so engineers spend their time building instead of babysitting a test suite.

There is also a difference in scope. Playwright is a foundational tool, a great engine and framework that you assemble everything else around. Shiplight is a full platform for the whole E2E concern: authoring and self-healing, hosted CI runners, dashboards, reporting, scheduling, and results, plus enterprise posture (SOC 2, VPC) and first-party support.

You are not on your own stitching infrastructure together and owning it when it breaks.

When Raw Playwright Alone Is Enough

Honest scope: some teams are better off staying with raw Playwright.

• Playwright is genuinely not your bottleneck. Very strong engineering teams with heavy existing investment, stable UIs, and low selector churn sometimes have E2E fully under control. If maintenance is not eating meaningful engineering time, Shiplight has nothing to remove.
• You need tests in a specific language like Python, Java, or .NET. Playwright serves those languages natively; Shiplight's authored artifact is language-neutral YAML. If the test language itself is a hard requirement, raw Playwright is the fit.
• You want a zero-vendor, fully open-source stack. That is a real constraint for some teams, and raw Playwright meets it where a managed platform cannot.

Already Using Playwright? Adoption Is Incremental

If you have a Playwright suite today, you keep it. Shiplight is Playwright-compatible and runs alongside an existing suite: reuse your config, keep every test you have, point Shiplight at the new and hard tests first, and let coverage grow from there. Nothing about the model requires a migration to start, and no test you already trust needs to move.

Starting fresh is even simpler: point Shiplight at your app and let the agent author the first tests. There is no Playwright suite to stand up first.

The strongest fit is teams already shipping with AI coding agents. An agent that writes a feature but cannot see a browser cannot close its own loop. Wiring Shiplight in over MCP means the agent verifies UI changes as it builds and leaves a regression test behind as a byproduct, so coverage stops being a separate project.

Final Verdict

Playwright is the best open-source browser engine for E2E testing, which is why Shiplight runs on it rather than competing with it.

Raw Playwright is the right call for teams with the engineering strength and suite stability to keep maintenance cheap, teams that need tests in Python, Java, or .NET, and teams for whom a zero-vendor stack is the binding constraint.

Shiplight is the stronger setup when selector maintenance is consuming real engineering time, when your team ships with AI coding agents that should verify their own changes, and when you want tests the whole team can read and a platform that handles the rest, without owning a pile of selector-bound test code and without giving up the Playwright engine underneath.

Book a demo, and bring your current Playwright suite if you have one, since the comparison is most useful against your own tests.

Frequently Asked Questions

References

Playwright Documentation, Model Context Protocol