The moment your product generates its output instead of computing it, expect(text).toBe('Your order shipped on March 3') stops being a test and becomes a coin flip. The fix is not to give up on assertions. It is to assert on the properties that survive a rewording and break on a regression.
This is a gap in the tooling conversation. Browser automation frameworks document how to drive a page, not what to check when the page's content is generated. Model evaluation platforms score a model against a fixed dataset offline, a different question from whether the shipped feature works in a browser for a logged-in user. Neither covers the middle, where your feature lives.
Why the normal assertion breaks
A deterministic feature has one correct output per input, so the test writes it down. A generative feature has a large space of acceptable outputs. "Your order shipped on 3 March", "Shipped March 3rd", and "It went out on the 3rd" are all correct. An exact-match assertion fails on two of the three.
Teams reach for three bad workarounds. Loosen until green: assert the response is non-empty, and the test now passes when the model returns "I'm sorry, I can't help with that", the regression you needed to catch. Pin the model and hope: temperature zero and a pinned version reduce variance without removing it, providers change models behind stable names, and your retrieval context moves as your data moves. Test the API and skip the UI: the most common one, and it misses every bug where the model is fine and the wiring is not.
The question a developer is actually asking
Staring at a red build, nobody wants a philosophical answer about non-determinism. The question is narrow: is this a real regression, or did the model just word it differently? Every technique below makes that answerable in one glance, by asserting the properties a rewording preserves and a genuine break destroys.
What to assert instead
Treat this as a menu. A serious AI feature test asserts four to six of these, not one.
Structure. If the output feeds a component, the shape is contractual even when the prose is not. A JSON response parses, has the required keys, uses a value from a known enum, has an array with at least one element. If your summariser returns markdown, assert a heading and three list items. Shape regressions are the most common real breakage and the easiest to assert on.
Presence of required elements in the rendered UI. Not the raw completion, the DOM. The answer block rendered, a citation chip appeared, the copy button exists, the follow-ups rendered. A model change that drops the structured section your renderer keys off is invisible at the API layer and obvious here.
Absence of forbidden content. The strongest assertions live here, because forbidden is a much tighter set than correct. Assert the output contains no refusal or apology strings ("I'm sorry", "As an AI", "I cannot"), no raw prompt leakage, no unrendered markdown fences, no placeholders like {{customer_name}}, no undefined printed as text, no other tenant's data, no unredacted PII. This list catches more real regressions than any content-quality check.
Grounding against known input. Seed a fact only your fixture contains, then assert it appears. Ask about an order whose seeded total is $412.50 and assert 412.50 is in the answer. A rewording preserves the number; a broken retrieval or a truncated context does not.
Latency budgets. Two numbers that fail differently. Time to first token tells you the request reached the provider and streaming started, catching auth failures, cold starts and proxy misconfiguration. Total completion time catches runaway generation. Assert both.
Streaming completes, and citations resolve. Both covered in full below. A stream that dies at 80% looks plausible in a screenshot and is broken for the user, and citation resolution is the highest-value RAG assertion by a distance.
Error and refusal states. Deterministic and fully testable, which makes them the cheapest coverage you will ever add. Force a provider 500, a timeout, a rate limit, a content-filter block and an empty retrieval, then assert the UI shows the right message, offers a retry, and leaves no half-rendered answer beside an error.
Token and cost ceilings. Assert the request stayed under a token bound. This catches prompt-template bugs that inject the whole conversation history or an entire document into every call, which surfaces as a bill rather than a failure. If token counts are exposed in a header or debug endpoint, it is a one-line assertion with a high catch rate.
Testing streaming UI and partial renders
Streaming breaks in ways nothing else does, because the interface passes through a sequence of intermediate states and each one can be wrong on its own. What to check, in the order these bite:
- The first chunk renders. Not the final answer, the first visible token. The real "the feature is alive" assertion, and it fails fast when the connection is wrong.
- Text grows monotonically. Sample the answer container two or three times mid-stream and assert length increased. Catches a re-render that wipes the buffer and restarts.
- Markdown does not break mid-parse. A partial code fence or half-written table renders as garbage in many renderers. Assert no raw fence characters appear at any sampled point and the layout held.
- The terminal state arrives. Typing cursor gone, loading class removed, copy and regenerate enabled. Assert on the affordance rather than the text, because it is deterministic.
- Abort works. Click stop mid-stream and assert text stops growing, the request is cancelled, and the partial answer is kept or cleared as your product intends. Almost nobody tests this and it breaks often.
RAG-specific checks
Retrieval features have failure modes a generic content assertion never sees, and most are silent.
Does the citation actually resolve? The single most valuable check here. A model will produce a confident, correctly formatted citation to a document that does not exist, or to a real one that does not contain the claim. Do both halves: follow the link and assert a success status rather than a 404 or login redirect, then fetch the target and assert the quoted snippet is in it. A citation that renders but points nowhere manufactures trust, which is worse than no citation.
Did retrieval return anything at all? Assert the retrieved-document count is above zero for a question the corpus answers. An empty retrieval that falls back to the model's own knowledge reads fine and is completely ungrounded.
Does an out-of-corpus question get refused? Ask something your corpus cannot answer and assert the feature says it does not know. This is the one place you want a refusal string present rather than forbidden, which is why that list is per-test rather than global.
Is the index fresh? Seed a document, wait for your indexing path, ask a question only that document answers, and assert the new content appears. Catches a broken ingestion pipeline, which otherwise surfaces weeks later.
Does permission scoping hold? Log in as user A and ask a question whose answer sits in user B's document, then assert that content does not appear. This is the RAG bug that ends up in an incident report, and it is a five-line test.
How this differs from offline evals, and why you need both
Offline evaluation runs a fixed set of prompts against the model, scores the outputs, and tracks the score over time. That is the right instrument for a model-quality question: is the new model better, did the prompt rewrite help, which retrieval strategy wins.
It is the wrong instrument for "does the feature work in production", because it never touches your product. Every one of these passed an eval suite and shipped broken: a prompt template deployed with a variable unsubstituted, so users saw {{user_name}}; a UI change that truncated context before the request left the browser; a streaming parser broken by an SDK upgrade; citation links rendered as relative paths that 404'd; the feature flag off in production; the auth token not forwarded to the retrieval service, so every retrieval returned empty and the model answered from memory.
Evals score the brain. Browser-level tests check the brain is connected to the body. Run evals when the model or prompt changes, and feature tests on every deploy.
Test data and seeding
AI features are unusually sensitive to fixtures, because the input space includes your data.
Seed a small, fixed corpus. Ten to thirty documents with facts you control, including a couple of awkward ones (a table, another language, one nearly empty). Every grounding assertion keys off it. Never ground assertions in production data, which changes underneath you and makes every failure ambiguous.
Keep a golden question set. Fifteen to forty questions with the property each answer must have, not the answer text: "must contain 412.50", "must cite doc-7", "must refuse". That artefact keeps the suite maintainable, because it separates what to check from the mechanics of checking it.
Decide per suite whether the model is real. Recorded or stubbed provider responses give a deterministic test of rendering, streaming, error handling and citation resolution, fast and variance-free enough for every pull request. A nightly live-model suite catches provider drift and prompt regressions. Pick only one and you get either a suite that cannot fail for the right reason or a suite nobody trusts.
Seed accounts and state, do not click to create them. Setup here is heavy (an account, a tier that enables the feature, conversation history, an indexed corpus), so build it through an API or fixture and save the browser for the part you are testing.
Model flakiness versus a genuine bug
Both look like an intermittent red build. Telling them apart is a procedure: rerun ten times and read the pass rate.
- Zero of ten. Not the model. Something is broken, and the wording is a distraction.
- Eight or nine of ten. Your assertion is too tight for the acceptable output space. Move from exact text to a property: from "shipped on March 3" to "contains the seeded order number and no refusal string".
- Four or five of ten. The band people misread as flakiness. A coin flip is rarely phrasing variance. It is usually a real intermittent bug: a race between the stream and the render, a retrieval timeout under load, a cache warm half the time.
Two habits make this cheap. Log the output on failure with the prompt, model version and retrieved document ids, because an AI test failure without the output is unreadable. And track flake rate per assertion rather than per test, since it is almost always one over-tight assertion in a sound test.
If an assertion cannot be made stable, delete it rather than quarantining the whole test. Five solid property assertions with one removed beats a skipped test, and beats one loosened until it cannot fail.
A starting suite
Six tests, in order: grounded happy path, citation resolution, streaming lifecycle, forced provider 500, out-of-corpus refusal, permission scoping. Fold the forbidden-content and budget assertions into the first rather than writing them separately.
How Shiplight handles the non-deterministic part
Deterministic steps (click this, fill that, expect this element) cache their resolved locator, so on a hit they run at full speed with no model call. AI steps and AI assertions are not cached. They call the model every run, which is what lets them judge what a fixed selector cannot: whether the answer panel shows a real response rather than an error, whether the summary refers to the invoice the test seeded. That is a per-run cost and the honest trade, so never assume a whole test is cached.
The tests are readable YAML in your own git repository, so an AI feature's property list (seeded number present, no refusal, a rendered citation) can be reviewed by the person who knows what the feature should do. When the application is genuinely broken, the fix workflow reports the bug rather than editing the test to pass.
Scope limits, plainly: Shiplight covers web applications, so native mobile and desktop are outside it, and executing a test file needs an LLM key, ours or your own, even though authoring runs through your coding agent's subscription with no Shiplight account.
FAQ
How do you write an assertion when the output changes every time?
Assert on properties the wording cannot change: structure, a seeded fact, required UI elements, absence of forbidden strings, latency and token budgets. Four or five of those in one test give a signal an exact-match assertion never could. Save exact matching for the genuinely deterministic parts, such as error and refusal states.
Do I still need offline evals if I test the feature in a browser?
Yes, they answer different questions. Evals score model and prompt quality against a fixed dataset; browser tests confirm the shipped feature is wired up, rendering, streaming and grounded. Most production AI incidents are wiring failures an eval suite cannot see.
How do I test a streaming response?
Check the sequence rather than only the end state: the first chunk renders quickly, text grows monotonically when sampled mid-stream, no broken markdown appears, and the terminal state arrives with controls re-enabled. Also test abort, which breaks often and is almost never covered.
What is the most valuable check for a RAG feature?
Whether the citation resolves. Follow the link, assert it is not a 404 or a login redirect, then fetch the target and assert the quoted snippet is in it. A confident citation to a document that does not support the claim is what costs you user trust fastest.
Is a failing AI test a bug or just the model being different?
Rerun it ten times. Zero of ten means something is broken, eight or nine of ten usually means your assertion is too tight, and four or five of ten is normally a real intermittent bug such as a render race or a retrieval timeout. Always log the output, model version and retrieved document ids on failure.
Should tests call the real model or a stub?
Both, on different cadences. Stubbed responses make rendering, streaming, error handling and citation logic deterministic and cheap enough for every pull request. A smaller live-model suite nightly catches provider drift and prompt regressions.
