Flaky Test

In one sentence

A flaky test is a test that gives different results — pass or fail — for the same code, depending on factors unrelated to whether the code is correct: timing, network conditions, test ordering, shared state, or non-deterministic resolution.

Why flakiness is worse than no test

A test that always fails is a clear signal: fix it. A test that always passes is also clear: trust it. A flaky test is the worst case — it sometimes catches real bugs, sometimes fires false alarms, and engineers learn to ignore it. The signal-to-noise ratio of the entire suite collapses, and even valid failures get re-run away.

Eight common root causes

1. Timing assumptions — fixed sleeps, missing waits for async work to settle.
2. Test ordering — tests sharing state that's only clean in a specific order.
3. External dependencies — network calls, third-party APIs, real email or auth services.
4. Concurrency — parallel test workers fighting over the same database row or DOM state.
5. Non-deterministic UI resolution — selectors that match different elements depending on render order.
6. Environment drift — staging data changes between runs.
7. Resource exhaustion — memory or connection limits hit under load.
8. Improper resource management — leaked browser contexts, unclosed connections, lingering processes.

How flakiness compounds in AI-velocity teams

When AI coding agents open 40+ pull requests per week, even a 1% flake rate produces multiple false failures every day. Engineers develop "rerun reflex" instead of debugging — and real regressions slip through.

What to do

• Quarantine, don't delete — see quarantine test.
• Measure flakiness as a first-class metric — flake rate by test age (newer tests are flakier; older tests should not regress).
• Track a test flakiness budget — like an SRE error budget, but for the testing layer.
• Fix the root cause — re-running until green hides the problem.