Test Data Management: The Unglamorous Reason Suites Fail

A test that passes alone and fails in the suite is almost never a timing bug. It is usually two tests touching the same record, in an order that changes when the runner parallelizes.

Test data is the least discussed part of test automation and one of the largest sources of the flakiness that eventually gets a suite disabled.

The problem

An end-to-end test needs the application in a particular state. A user who exists, a cart with items, an order in a specific status. Getting to that state is setup, and how you do it decides whether the suite survives contact with parallelism.

Three failure patterns account for most of it:

Shared mutable state. Two tests use the same account. One changes its subscription tier. The other now sees a different page than it expected. Both pass in isolation and one fails when they run together, which is exactly the signature people misdiagnose as timing.

Order dependence. Test B works because test A created something. Someone shards the suite across four machines, A and B land on different workers, and B fails for reasons that look like nothing to do with data.

Leftovers. A test creates records and does not clean up. The database drifts, and after two hundred runs a query returns more rows than the assertion expects.

All three get worse with parallelism, which is the thing everyone reaches for when the suite gets slow. Speeding up a suite with shared data is how a slow suite becomes an unreliable one.

The strategies, and what each costs

Per-test creation. Each test creates exactly what it needs, ideally through the API rather than the UI, and cleans up after. Slowest per test and by far the most reliable. This is the right default.

Unique identifiers everywhere. Never test@example.com. Use a unique value per run so two tests cannot collide even when they do the same thing. Cheap to adopt and eliminates a whole class of collisions on its own.

Fixtures for read-only reference data. Countries, product categories, plan definitions. Safe to share precisely because nothing writes to them.

Reset between runs. Restoring a known state before each run works well locally and gets expensive in CI. A middle path is resetting per suite rather than per test.

Seeded factories. Helpers that build a valid object with sensible defaults and let a test override the one field it cares about. This is what keeps per-test creation from becoming forty lines of setup.

The general rule: share only what nothing writes to. Everything mutable belongs to one test.

Production data

Copying production data into a test environment is tempting because it is realistic, and it is the fastest way to create a serious problem.

Real customer records in a test environment are real customer records, subject to the same obligations, in a system with weaker access controls and probably a public preview URL. Anonymization is harder than it sounds: names and emails are easy, but behavioral patterns and rare combinations re-identify people surprisingly well.

Synthetic data generated to match production's shape rather than its contents gets most of the realism without the exposure. If you genuinely need production data, subset it, anonymize it properly, and treat that environment with production-grade access control.

What changes with agent-written tests

Agents generating tests will generate setup too, and they tend toward the pattern that works right now rather than the one that survives parallelism. Hardcoded emails, assumptions that a record exists, no cleanup. Each test passes when written, and the suite degrades as it grows.

This is worth catching early, because it is much cheaper to establish the convention than to retrofit two hundred tests. Two things help:

Make the convention binding, not advisory. A hook or a lint rule that rejects hardcoded identifiers holds where a note in a prompt does not. See Claude Code skills for where each layer belongs.

Give the agent factories to use. An agent with a createUser() helper available will use it. An agent without one will write a hardcoded fixture, correctly, because you gave it nothing better.

Shiplight generates tests as intent-based YAML committed to your repository, which matters here for one specific reason: the setup is visible in the diff and reviewable, rather than held in a vendor's cloud where nobody notices the hardcoded account until the suite starts failing. Web applications; native mobile is not in it.

Frequently Asked Questions

Related Reading

• How to fix flaky tests: the symptom this usually causes
• QA automation: where maintenance cost accumulates
• End-to-end testing: the layer with the hardest setup
• AI testing data security: handling sensitive data in test environments