--- title: "Test Data Management: The Unglamorous Reason Suites Fail" excerpt: "Most flaky end-to-end tests are not timing problems. They are data problems: tests that assume a record exists, or leave one behind that breaks the next run." metaDescription: "Test data management explained: why shared test data causes flakiness, strategies from fixtures to per-test setup, and handling production-like data safely." publishedAt: 2026-07-30 author: Shiplight AI Team categories: - Guides - Best Practices tags: - test-data-management - test-automation - flaky-tests - e2e-testing - qa-process metaTitle: "Test Data Management: Why Suites Fail" featuredImage: ./cover.png featuredImageAlt: "Illustrated Shiplight blog cover: isolated data sets feeding separate test runs, with one shared pool shown causing collisions." --- A test that passes alone and fails in the suite is almost never a timing bug. It is usually two tests touching the same record, in an order that changes when the runner parallelizes. Test data is the least discussed part of test automation and one of the largest sources of the flakiness that eventually gets a suite disabled. ## The problem An end-to-end test needs the application in a particular state. A user who exists, a cart with items, an order in a specific status. Getting to that state is setup, and how you do it decides whether the suite survives contact with parallelism. Three failure patterns account for most of it: **Shared mutable state.** Two tests use the same account. One changes its subscription tier. The other now sees a different page than it expected. Both pass in isolation and one fails when they run together, which is exactly the signature people misdiagnose as timing. **Order dependence.** Test B works because test A created something. Someone shards the suite across four machines, A and B land on different workers, and B fails for reasons that look like nothing to do with data. **Leftovers.** A test creates records and does not clean up. The database drifts, and after two hundred runs a query returns more rows than the assertion expects. All three get worse with parallelism, which is the thing everyone reaches for when the suite gets slow. Speeding up a suite with shared data is how a slow suite becomes an unreliable one. ## The strategies, and what each costs **Per-test creation.** Each test creates exactly what it needs, ideally through the API rather than the UI, and cleans up after. Slowest per test and by far the most reliable. This is the right default. **Unique identifiers everywhere.** Never `test@example.com`. Use a unique value per run so two tests cannot collide even when they do the same thing. Cheap to adopt and eliminates a whole class of collisions on its own. **Fixtures for read-only reference data.** Countries, product categories, plan definitions. Safe to share precisely because nothing writes to them. **Reset between runs.** Restoring a known state before each run works well locally and gets expensive in CI. A middle path is resetting per suite rather than per test. **Seeded factories.** Helpers that build a valid object with sensible defaults and let a test override the one field it cares about. This is what keeps per-test creation from becoming forty lines of setup. The general rule: **share only what nothing writes to.** Everything mutable belongs to one test. ## Production data Copying production data into a test environment is tempting because it is realistic, and it is the fastest way to create a serious problem. Real customer records in a test environment are real customer records, subject to the same obligations, in a system with weaker access controls and probably a public preview URL. Anonymization is harder than it sounds: names and emails are easy, but behavioral patterns and rare combinations re-identify people surprisingly well. Synthetic data generated to match production's *shape* rather than its contents gets most of the realism without the exposure. If you genuinely need production data, subset it, anonymize it properly, and treat that environment with production-grade access control. ## What changes with agent-written tests Agents generating tests will generate setup too, and they tend toward the pattern that works right now rather than the one that survives parallelism. Hardcoded emails, assumptions that a record exists, no cleanup. Each test passes when written, and the suite degrades as it grows. This is worth catching early, because it is much cheaper to establish the convention than to retrofit two hundred tests. Two things help: **Make the convention binding, not advisory.** A hook or a lint rule that rejects hardcoded identifiers holds where a note in a prompt does not. See [Claude Code skills](/blog/claude-code-skills) for where each layer belongs. **Give the agent factories to use.** An agent with a `createUser()` helper available will use it. An agent without one will write a hardcoded fixture, correctly, because you gave it nothing better. [Shiplight](/plugins) generates tests as intent-based YAML committed to your repository, which matters here for one specific reason: the setup is visible in the diff and reviewable, rather than held in a vendor's cloud where nobody notices the hardcoded account until the suite starts failing. Web applications; native mobile is not in it. ## Frequently Asked Questions ### Why do my tests pass alone and fail in the suite? Almost always shared mutable data. Two tests touch the same record and the order changes when the runner parallelizes. ### Should each test create its own data? Yes, as a default. It is slower per test and far more reliable, and factories keep the setup short. ### Can I use production data in tests? Prefer synthetic data matching production's shape. Real records carry the same obligations in an environment with weaker controls, and anonymization is harder than it looks. ### What is the cheapest improvement? Unique identifiers per run. Never a fixed test email. It removes a whole class of collisions for almost no effort. ### How does this change with AI-generated tests? Agents write setup that works now rather than setup that survives parallelism. Give them factories and make the convention binding with a hook or lint rule. ## Related Reading - [How to fix flaky tests](/blog/how-to-fix-flaky-tests): the symptom this usually causes - [QA automation](/blog/qa-automation): where maintenance cost accumulates - [End-to-end testing](/blog/end-to-end-testing): the layer with the hardest setup - [AI testing data security](/blog/ai-testing-data-security-soc2): handling sensitive data in test environments