Tests Are the Best Context You Can Give a Coding Agent

The most useful context you can give a coding agent is not a longer prompt or a bigger design doc. It is a set of tests in the repo. Tests tell the agent, in executable terms, what "working" means for your product. They catch its mistakes the moment it makes them, and they let it correct itself without a human reading a single line of the diff. A prompt describes intent in prose the agent can misread. A test asserts intent in a form the agent can run, fail, and respond to. That difference is the whole game.

I run an AI-native testing company, so I have a stake in this. But the argument holds independent of any product: when an agent has to guess whether its change worked, it guesses confidently and often wrong. When it can run a test that encodes the expected behavior, the guessing stops. The test is the ground truth, and the agent is the thing that keeps editing until the ground truth is satisfied.

Why a test outperforms a prompt as context

Context engineering for coding agents is the discipline of deciding what an agent sees before it acts. Most of that conversation is about documents: system prompts, architecture notes, style guides, retrieved snippets. Those help the agent form a plan. They do not tell it whether the plan worked.

A test does. Prose context is feedforward: it points the agent in a direction before it starts. A test is feedback: it observes what the agent actually produced and reports back. Birgitta Böckeler, writing on Martin Fowler's site about harness engineering for coding agents, frames tests as "sensors" that "observe after the agent acts and help it self-correct." A prompt cannot do that. A prompt is a hope. A failing test is a fact.

This is why test-driven development has become the strongest pattern for agentic coding. Anthropic's own Claude Code best practices say it directly: each red-to-green cycle gives the agent unambiguous feedback. Write the test first, let it fail, then let the agent implement until it passes. The agent is no longer inventing its own definition of done. You gave it one, in code, and it cannot argue with a red result. As Addy Osmani puts it in his writing on self-improving coding agents, "without checks, an autonomous agent might merrily introduce bugs or failing builds while thinking it succeeded." Take the checks away and the agent's confidence and its correctness stop being correlated.

What the loop looks like in practice

Here is the concrete shape, the thing that happens dozens of times in a single agent session:

1. The agent reads the existing tests for the area it is about to change, so it knows what behavior is load-bearing before it touches anything.
2. It writes the code and runs the tests.
3. A test fails, naming the expected behavior, the actual behavior, and where they diverged.
4. The agent reads that, forms a new hypothesis, edits, and runs again.
5. It repeats until green, then moves on, having verified its own work.

No human sat in that loop. That is the point: the agent runs longer without supervision because it has a fast, honest signal at every step. Owain Lewis, writing about agent feedback loops, calls a good harness the thing that "self-corrects as many issues as possible before they even reach human eyes." Tests are the load-bearing part of that harness.

Unit tests already do this well for logic, and agents are good at generating and consuming them. The loop breaks down at the UI. A unit test passes while the button renders off-screen; a type check passes while the modal traps focus. The behavior the user experiences lives in the rendered app, and most agents have no way to observe it, so they verify the logic, declare success, and ship a broken screen. That is not an agent failure but a missing sensor: it could not see what broke, so it could not test or correct it.

What makes a test good context, and what makes it useless

Not all tests are usable context. A test the agent cannot read, run cheaply, or maintain becomes a liability the moment the code around it moves. Three properties separate the two.

Readable, so the agent can reason about intent. A test written as a brittle chain of CSS selectors tells the agent what the DOM was, not what the user was trying to do. When the agent reads a test that states intent, log in, add an item, confirm the total updates, it understands the contract and can preserve it while changing the implementation underneath. This is the case I made in YAML-based testing: tests authored from intent stay legible to both humans and agents.

In the repo, so the context travels with the code. Context that lives in a vendor's cloud is invisible to the agent working in your checkout. Tests that live in your git history sit next to the code, versioned with it, readable by any agent you point at the repo. When Tuesday's agent ships a change and Thursday's agent has to modify the same flow, the test is the durable memory between them. This is the larger point in the testing layer for AI coding agents: the artifact has to outlive the session that created it, or you pay the verification cost fresh every time.

Self-healing, so maintenance does not eat the agent. The honest objection to "just add more tests" is that tests break when the UI legitimately changes, and something has to fix them. If that something is a human, you have moved the bottleneck, not removed it. If the fix is a silent rewrite in a vendor's cloud, you have traded a flaky test for an untrustworthy one. The workable version: the agent repairs the test in a real browser and surfaces the heal as a reviewable diff in the PR, so a person approves the change instead of authoring it. I went deeper on that in can coding agents test their own code and on the mechanics in coding-agent plugins and automated test generation.

Those three properties describe context an agent can consume and maintain, not just read once. This is where Shiplight sits. It plugs into the coding agent as an MCP server, gives it eyes and hands in a real browser to verify UI changes as it builds, and turns those verifications into readable YAML tests that live in your repo and run locally with npx shiplight test. Three commands carry the loop: /shiplight verify confirms a change looks right after an edit, /shiplight create-yaml-tests has the agent walk the app and write E2E coverage, and /shiplight fix reproduces failures and root-causes them, reporting a real bug instead of quietly editing the test when the app is what broke. The tests are Playwright-compatible, so adoption is additive, not a rewrite. At HeyGen, the head of QA's team went from spending most of its time maintaining Playwright tests to close to zero within a month, because the agents started leaving durable, verifiable tests behind them instead of nothing.

Where this is weak: the cold start

The honest limit is the empty repo. Tests are the best context you can give an agent, but a greenfield project has none, and neither does a legacy codebase that was never tested. On day one there is no ground truth to check against, so the agent is back to guessing.

There is no clever escape from this. You have to seed the loop, and the good news is that seeding is now cheap: pointing an agent at the running app to walk the critical flows produces a first suite in hours instead of weeks, and teams routinely stand up their first few hundred E2E tests in the opening week. But the first tests still need a human to confirm they encode the right intent. A test written against buggy behavior faithfully protects the bug. Verification confirms intent; it cannot invent it, and it cannot tell you the intent was wrong.

The other limit is judgment. A test can confirm checkout completes, but not that the new checkout flow is worse for users. Tests make the agent trustworthy on "does it work" and say nothing about "is it good." Keep a human on that question and on the merge button, and let the tests handle the part that scales.

Frequently Asked Questions

How do I start using tests as context for AI agents?

Give the agent the existing tests for the area it is changing, and make sure it can run them. Before it edits, the tests tell it which behaviors are load-bearing; after it edits, running them tells it whether it broke anything. The agent reads a failure, fixes the code, and reruns until green, without a human in the loop. For UI work it also needs a way to observe the rendered app, since logic-only tests miss visual and interaction regressions.

Why are tests better context than a detailed prompt?

A prompt is feedforward: it points the agent in a direction but never tells it whether the result was right. A test is feedback: it reports the gap between expected and actual. The agent can misread prose and proceed confidently. It cannot argue with a failing assertion. That is why test-driven development has become the strongest pattern for agentic coding.

Can a coding agent write its own tests to use as context?

Yes, and it is the practical way to seed the loop. An agent can walk a running app and generate E2E coverage in hours, which solves the cold-start problem of an untested repo. The one guardrail: a test written against buggy behavior protects the bug, so a human should confirm the first tests encode the intended behavior. After that the agent can maintain and extend them on its own.

What makes a test usable as agent context instead of a liability?

Three things: it should read as intent rather than brittle selectors, so the agent understands the contract; it should live in the repo, so it travels with the code across sessions; and it should self-heal when the UI legitimately changes, surfacing the repair as a reviewable diff rather than forcing a human to rewrite it. Tests missing those properties break faster than they help.

Where does using tests as context break down?

At the cold start and at judgment. An untested repo has no ground truth until you seed a suite. And a passing test confirms behavior, not quality: it verifies checkout completes but cannot tell you the flow is worse for users. Keep people on the initial intent and the product call, and let the tests handle the part that scales.