Every testing tool makes one decision early that is very hard to reverse: what a test physically is. Either it is a file in your version control system, or it is a row in the vendor's database with a web form on top of it.
That decision looks like a storage detail. It is not. It determines whether your tests participate in the workflow your engineering team already runs, or sit next to it in a separate system with its own login, its own history, and its own idea of what the current state of the world is.
The five things a row cannot do
A row does not diff. When a test changes, the useful question is what changed and why. In a console, the answer is an audit log entry: "Sarah edited Checkout Flow, 14:22." You cannot see that the assertion on the order total was replaced with an assertion that the page loaded. A file gives you the before and after, side by side, permanently, attached to the commit that did it.
A row does not appear in code review. This is the expensive one. A pull request changes the checkout flow. The reviewer sees the application diff. The tests that cover that flow are in another system, and nothing in the review surfaces the fact that they were edited yesterday to accommodate the change. Review is where teams catch a weakened test, and a weakened test is exactly what nobody catches when the test lives elsewhere. When the test is a file, changing it is part of the diff, and a reviewer who sees an assertion disappear can ask about it in the same thread as the code.
A row does not branch. Feature branches exist because a change and its consequences travel together until they are ready. Tests in a database have one state: current. So a test updated for an unreleased feature is now failing on main, and a test for the current release is now wrong on the branch. Teams work around this with naming conventions, disabled flags, and environment tags, which is a re-implementation of branching by hand, done worse. Tests as files branch for free, because they are in the branch.
A row does not roll back. Reverting a bad release should revert what verified it. With files, git revert takes the tests with the code, so the suite that runs afterwards matches the deployed commit. With rows, the code goes back and the tests do not, and the first post-revert run is red for reasons unrelated to the bug you were escaping.
A row is not editable by your agent. This is the newest and increasingly the most consequential. A coding agent working in your repository can open a test file, read it, understand what it asserts, change it alongside the code, and include it in the same commit. It cannot do any of that with a record behind a web console, and the workaround, an API the agent calls to mutate the record, is not equivalent: the agent is now issuing writes into a system it cannot read back in full, cannot diff, and cannot reason about as a whole. Ask what happens when the agent needs to know whether a flow is already covered before writing a new test. Against files, it greps. Against a console, it cannot.
And there is a sixth, which is about the end rather than the day-to-day. Leaving means rewriting. If your tests are rows in a vendor's schema, the thing you own on the way out is an export file, if there is one, in a format nothing else runs. The switching cost is not the migration project, it is the entire re-authoring of your coverage, and every team that has priced that has concluded it is cheaper to stay. That is not an accident of the architecture. It is the commercial function of it.
The counter-arguments, taken seriously
There are real reasons console-based platforms exist, and a version of this argument that ignores them is not worth reading.
Non-engineers need to edit tests. This is genuine. In plenty of organisations the person who knows what the checkout flow should do is a QA lead or a support engineer without a git workflow, and telling them to open a pull request is telling them not to contribute. A console is a legitimate answer to that problem.
But the answer is not the only one, and it has become weaker. Two things changed. First, tests written in readable structured text rather than code are editable by anyone who can read the flow, which moves the barrier from "can you write code" to "can you use a web editor on a file", and every git host has one. Second, the person who cannot write a test can now describe what they want to a coding agent, which writes it into the repo and opens the pull request for them. The non-engineer path no longer requires a separate database.
The console genuinely gives you things a repo does not. Also true, and worth stating plainly. Run history over months, flake rates per test, trend lines, dashboards a VP can read, cross-run comparison, retention of screenshots and videos in a browsable form. A git repository stores none of that well, and pretending otherwise is how you end up with a team writing custom report parsers.
The correction is not to give the reporting up. It is to notice that reporting is downstream of runs, not upstream of tests. Run results are a database problem: append-only, high volume, queried in aggregate. Test definitions are a version control problem: edited, reviewed, branched, reverted. Storing both in the same place because one of them belongs there is the error.
Hosted execution is genuinely useful. Nobody wants to maintain a fleet of browser runners, and self-hosted CI for browser tests degrades in annoying ways: browser versions drifting, memory limits, parallelism that costs real money to get right. A tool without hosted runners is asking every customer to solve that themselves.
A repo full of tests nobody reviews is not better. Fair. Putting tests in git does not automatically produce review discipline, and a team that rubber-stamps test diffs gets no benefit from their location. The claim is that files make the good practice available, not that they enforce it.
The distinction that actually matters
Almost every comparison in this category conflates two independent questions:
- Where does the artefact live? The test definition: the thing you edit, review, and own.
- Where does execution happen? The runners: the machines that open browsers and produce results.
These are separable, and treating them as one axis is what makes the debate feel like a tradeoff when it is not.
| Artefact in your repo | Artefact in the vendor's database | |
|---|---|---|
| Runs on your machines and your CI | Full ownership, and you own the runner problem too | Uncommon in practice: if the definition lives there, execution usually has to as well |
| Runs on hosted runners | Ownership of the artefact, someone else's operational problem | The classic console platform: convenient, and the exit is a rewrite |
The top-right cell is the one worth wanting, and the bottom-right is the one most teams end up in because the two questions were bundled at purchase. A tool that keeps the test in your repository and offers hosted execution gives you both: the artefact is yours, versioned and reviewable, and the runners are somebody else's job. The reporting sits where reporting belongs, over the run results, and it does not need custody of your test definitions to do its work.
The test for whether a vendor has genuinely separated these is simple, and it is a documentation question rather than a sales question. Can you read the test in a text editor, in a format that is documented? Does it live in your repository rather than being synced to a copy there? And if you stopped paying, what do you still have that runs?
What "an exit exists" has to mean
Portability claims are cheap, so it is worth being exact about what makes one real.
A proprietary format that only the vendor's runtime executes is not portable, even when it is stored as a file in your repo and even when there is an export button. A file you own that only one company's binary can run is a row with extra steps. The commercial dependency is identical; only the storage location moved.
An exit is real when the thing you can extract runs on something you do not buy from the vendor. In practice that means the format compiles down to, or converts into, an open standard with an independent implementation. For browser testing the honest end state is open-source framework code, because that is the thing that runs anywhere, on any CI, for free, with no account.
It is a five-minute check, and better done before you buy: find the public documentation for the format and for the export or transpilation, then see whether the output is something with its own community. If neither exists in the docs, the exit is a conversation with an account manager, which is not an exit.
How Shiplight handles both sides
Tests are readable YAML files that live in your git repository, alongside the code they cover. They diff, they appear in pull requests, they branch with features, and they revert with a revert. Your coding agent reads and edits them directly as files, which is the mechanism behind agents authoring and maintaining coverage rather than a person doing it in a console.
At run time each YAML test transpiles to Playwright plus Shiplight's own SDK code. That is the exit path, and it is a real one: the executable form is open-source framework code, which is also why every Playwright browser and feature is available. You do not need to know Playwright to use Shiplight or to write a test, and nothing about the workflow asks you to. It matters for one question only, the one about what you still have if you leave.
The hosted side is not missing, and this is the part comparisons often get backwards. There are Shiplight-hosted CI runners with browsers pre-installed, a cloud dashboard with run history, trend tracking and flaky-test detection, and an LLM proxy so tests that use AI steps do not need you to manage model credentials. You can also run on your own CI with your own LLM key from any major provider, on any plan including the free one. Both modes, your choice, is the point: infrastructure is optionality, and having the hosted option is a strength as long as it is not the only option.
Two things worth being precise about. Deterministic steps cache their resolved locator and run at full browser speed with no model call, but AI steps and AI assertions are not cached, because calling the model every run is exactly what lets them handle what a fixed selector cannot. And authoring through your coding agent runs on that agent's own subscription with no Shiplight account needed, while executing a test file needs an LLM key, ours or yours.
A checklist for the buying conversation
Five questions, all answerable from public documentation:
- Can I open a test in a text editor? If the answer involves logging in, the artefact is a row.
- Does a test change show up in a pull request? Not "can it be synced", but is the repo the source of truth.
- Can my coding agent edit tests directly as files? If the documented path is "the agent calls our API" or, worse, "the agent should not edit the files", authoring is vendor-mediated.
- Do tests branch with my code? Ask what happens to a test edited for an unreleased feature.
- What runs if I stop paying? The only good answers name a format with an independent implementation.
None of these is about features. They are about which system owns the artefact, and that decision outlives most of the feature comparisons made alongside it.
FAQ
Why should tests live in the same repository as the code?
Because they change for the same reasons and at the same time. Keeping them together means a behaviour change and its test arrive in one reviewable diff, branch together, and revert together. Split them across two systems and the two states drift, usually silently.
Can a coding agent maintain tests stored in a vendor console?
Not in the way it maintains files. An agent can call an API to mutate records, but it cannot read the whole suite, diff a change, or include the edit in the same commit as the code. Whenever the documented path for an agent is a set of vendor tools rather than the files themselves, authoring is vendor-mediated by design.
Do non-engineers need a console to edit tests?
They need a readable format and an easy editor, which a console is one way to provide. Tests written as plain structured text can be edited in any git host's web editor, and a non-engineer can also describe the change to a coding agent, which writes it and opens the pull request. See YAML-based testing for what that format looks like.
Is hosted execution bad if tests live in my repo?
No, and the two are not in tension. Where the artefact lives and where it runs are separate decisions, and hosted runners solve a real operational problem. What matters is that the hosted option is an option, so you can also run on your own CI without rewriting anything.
What makes a portability claim real rather than marketing?
That the extracted output runs on something you do not buy from the vendor. A proprietary format stored in your repo but executable only on the vendor's runtime carries the same commercial dependency as a database row. Look for a documented conversion to an open-source framework with an independent implementation.
Does storing tests in git give me run history and dashboards?
Not on its own, and it should not try to. Run results are append-only, high volume, and queried in aggregate, which is a database problem; test definitions are edited, reviewed and branched, which is a version control problem. A good platform puts each in the right place rather than using one as an excuse to own the other.
