What Does an Agent-Run Test Suite Actually Cost Compared to Plain CI?

On the bill, an agent-driven suite costs plain CI compute plus model tokens, and the token line is set by how many AI steps your tests contain, not by how many tests you have. At a typical density of one or two AI steps per test, expect roughly two to four times your current CI bill. Make every step and every assertion an AI call and yes, an order of magnitude is entirely achievable.

That is the short answer to a question asked on Hacker News in almost exactly those words: is this an order of magnitude more expensive than my existing CI running Playwright tests? It deserves arithmetic rather than reassurance, so this page works the numbers and marks which ones you have to measure yourself.

What plain CI costs today

Start with the baseline, because most teams have never itemised it.

Compute minutes. Hosted CI bills per minute of runner time, summed across jobs. A suite of 100 end-to-end tests averaging two minutes each is 200 minutes of billed runner time per full pass, regardless of how you shard it.

Parallelism does not change that number. Sharding across ten runners turns 200 sequential minutes into about 20 minutes of wall clock, and you still pay for 200 minutes. What sharding buys is feedback latency, which is worth real money in engineer attention but does not show up as a discount. The exception is a provider that bills per reserved parallel slot rather than per minute, where concurrency is exactly what you are paying for.

The retry tax. The line nobody budgets. A suite with a 5% flake rate and two automatic retries re-runs a slice of itself on most passes, which adds 10% to 20% to the minute count permanently and grows with the suite. Retries are also how a flaky suite hides: the bill goes up quietly instead of the build going red loudly.

The cost that never appears on the invoice. Someone reads the red builds and works out whether a failure is an application bug, a stale selector, a timing issue, or a bad fixture. Someone updates the selectors after a design change. QA engineers routinely report the majority of their automation time going to repair rather than to new coverage.

Put a number on it. If one engineer spends four hours a week on triage and test upkeep, at a fully loaded cost of around $120 an hour, that is about $2,000 a month. Against a compute bill of roughly $100 a month for the suite above. The invoice is 5% of the cost of the suite and the timesheet is the other 95%, which is the fact that makes the rest of this page make sense.

What an agent-driven suite costs

Two lines instead of one: the same compute, plus model tokens.

Compute is close to a wash. Shiplight bills compute at $0.012, $0.022, $0.042 and $0.082 per minute for small, medium, large and xlarge runners, which matches hosted-runner rates at every size. A test that took two minutes on your CI takes about two minutes on ours. If you run on your own runners instead, the compute line is literally unchanged, because it is the same line you already pay.

So the entire difference is tokens. Tokens get spent in two very different places, and conflating them is where most cost estimates go wrong.

Authoring is a one-off, per test. When your coding agent walks the application and writes the test, that conversation costs tokens. It happens once per test, it runs through your coding agent's own subscription over MCP, and it needs no Shiplight account and no charge from us. Amortised across every future run of that test, authoring rounds to nothing. It is not part of your CI bill.

Execution costs tokens only when a step calls a model. This is the variable that decides everything, and it is worth stating precisely rather than approximately.

The mechanic that decides the bill

A deterministic step resolves its locator once, and that resolution is cached. On a cache hit the step replays at full speed with no model call at all. Nothing about that step is AI-priced on a normal run.

AI steps and AI assertions are not cached. They call the model on every single run, by design, because calling the model is what lets them handle the thing a fixed selector genuinely cannot: a canvas region, a list whose contents change every run, a judgement like "the confirmation page shows the right order total".

Two consequences follow, and neither is intuitive.

First, a test is never "cached" as a unit. Steps are cached individually. A twenty-step test with two AI steps replays eighteen steps for free and pays for two, every run. Anyone telling you a whole test replays without touching a model is describing something else.

Second, per-run cost scales with AI step count, not test count. Adding your hundredth deterministic test adds compute minutes and nothing else. Adding one AI assertion to a test that runs on every pull request adds a model call to every pull request, forever. Those two additions look identical in a diff and behave completely differently on the invoice.

The third thing that calls a model is a heal: when the UI changed and a cached locator no longer resolves, the step is re-derived at run time from the stated intent. That is a real cost, and it is the cost that used to be a person's morning. It scales with how much your UI churns, which is the honest shape of maintenance cost in any tool.

The arithmetic, worked

Take the same suite: 100 tests, two minutes each, 200 minutes per full pass, 40 full passes a month.

Compute is $2.40 per pass at the small-runner rate. Now the token line. Assume an AI step costs about 10,000 tokens once you count the marked page context going in and the resolution coming back. Treat that 10,000 as a placeholder and replace it with your own measurement, because it varies with page complexity more than with anything else, and your run reports show the actual figure.

Model cost above uses the Standard tier at $3.00 per million tokens. Lite is $1.00 and Pro is $6.00, so the same 2M-token pass is $2.00, $6.00, or $12.00 depending on which tier that work needs.

At 40 passes a month, the two-AI-steps-per-test row is about $336 a month against about $96 of pure compute. So: three and a half times, not ten times, at a density most suites land on when someone is paying attention. The bottom row of that table is what an order of magnitude actually looks like, and it is reachable, and it is almost always a design choice rather than a requirement.

For scale: Pro is $60 a month and includes $60 of monthly credit, so the $336 suite spends its credit in the first week and runs pay-as-you-go after that. Free is $0 forever with a one-time $10 credit, which is enough to measure your own numbers before deciding anything.

Five ways to keep the run cost down

Prefer deterministic steps wherever the flow is stable. Login, navigation, form fills on a settled page. These are the majority of most suites and they should cost you compute and nothing else.

Reserve AI steps for what a fixed selector cannot do. Dynamic content, canvas and visual regions, anything where the correct answer requires judgement. If a stable data-testid would have worked, an AI step there is a subscription you are paying every run for a problem you did not have.

Watch AI assertions specifically. They are the easiest to sprinkle and the most expensive per run, because an assertion tends to sit on the hot path of every test. One AI assertion checking the end state usually beats five checking each intermediate screen, and reads better as a spec too.

Shard for latency, not for cost. Parallelism improves the number engineers feel and leaves the invoice alone. Size runners to the test rather than defaulting up: the xlarge rate is nearly seven times the small rate for the same minute.

Do not retry blindly. On a plain suite a blanket retry policy multiplies compute. On an agent-driven suite it multiplies compute and tokens, because the AI steps run again too. Retries are a reasonable tool for genuine infrastructure flakiness and a poor substitute for fixing a test that is telling you something.

Bring your own key, or use the hosted path

Both modes exist on every plan, including Free, and neither is an upgrade gate.

Point Shiplight at your own Anthropic, OpenAI or Google key, including Azure, Bedrock and Vertex, and your provider bills the tokens at whatever rate you have negotiated. Run the same YAML tests on your own CI runners and the compute line stays exactly where it is today. Do both and the arithmetic above collapses to your existing CI bill plus your existing model spend, with no metered vendor usage in the path at all.

The hosted path is a genuine product rather than a fallback: hosted runners, an LLM proxy, dashboards, artefact history and run evidence. What you give up by bringing your own everything is some of the usage and cost analytics, because those views are built from runs that pass through our runners or proxy. That is the trade, and it is the only one. Mixing is normal too: your key with our runners is a common shape.

The comparison that actually matters

A cheaper CI bill on a suite that needs a person babysitting it is not cheaper. Compare total cost of ownership over twelve months, on both sides:

compute minutes x rate
+ model tokens x rate
+ (engineer hours per week x loaded hourly cost x 52)
+ expected cost of regressions that reach production

For the suite above, plain CI is about $1,200 of compute a year against roughly $25,000 of engineer time at four hours a week. The agent-driven version is about $4,000 of compute and tokens a year. It is worth $2,800 more if, and only if, it moves the $25,000 term. Drop maintenance from four hours a week to one and it pays back about six times over. One Head of QA described going from roughly 60% of their time on authoring and repair to close to none within a month, on a suite of several hundred tests.

Work out your own number rather than taking ours. Six inputs, and five of them you already have: minutes per full pass, passes per month, current flake and retry rate, hours per week your team spends on triage and repair, and loaded hourly cost. The sixth is AI steps per test, and you set that yourself when you write the tests.

Where this does not pay off, stated plainly. If your existing suite is stable, nobody is babysitting it, and your UI does not churn much, the maintenance term you would be buying down is already small, and the token line is a real cost against a small saving. Teams with strong internal test platforms are often in exactly that position. The trade gets better as your interface changes faster, which is why teams shipping with coding agents feel it most sharply.

If you already have a Playwright suite, none of this is a migration. Tests transpile to Playwright at run time, so an agent-authored suite runs alongside what you have and you can add AI steps only where they earn their keep. If you have no suite today, you do not need to build a Playwright one first.

FAQ