Quarantine Test
Quarantining is the practice of removing a flaky test from the merge-blocking suite while keeping it running in a separate, non-blocking lane until the underlying flake is fixed. Quarantine preserves history and signal without letting flakiness corrupt the main pipeline.
In one sentence
Quarantining a test means moving it from the blocking lane (where it can fail a PR) to a non-blocking lane (where it still runs and reports, but doesn't gate merges) until its flake is diagnosed and fixed.
Why quarantine instead of delete or mute
Three reasons:
- Deletion loses history — the test still describes valuable behavior; deleting it forfeits that documentation and reintroduces the coverage gap.
- Muting (skipping) hides the problem — a
.skip()annotation often becomes permanent, and the team never circles back. Quarantine keeps the test running so the flake stays visible. - Signal preservation — quarantined tests can still catch genuine regressions; you just don't let them block merges in the meantime.
Quarantine workflow
| Step | Action |
|---|---|
| Identify | Test crosses a flake-rate threshold (e.g. >5% failures over 14 days against unchanged code) |
| Move | Tag the test or move it to a quarantined/ directory; pipeline stops blocking on it |
| Track | Open a ticket; assign an owner; budget engineering time |
| Fix | Address the root cause — see flaky test |
| Reinstate | Move back into the blocking lane after N consecutive green runs |
Common pitfalls
- No reinstatement bar — without a clear "exit criteria", quarantine becomes permanent muting in disguise.
- Quarantining real regressions — sometimes a "flaky" test is actually catching a genuine intermittent bug. Always investigate before quarantining; flake-rate alone is not proof of flakiness.
- No ownership — unowned quarantined tests rot. Assign every quarantined test a single human owner with a deadline.
Production rule of thumb
Quarantine within 24 hours of a flake's third occurrence. Reinstate after 30 consecutive successful runs across the typical PR mix. This prevents both knee-jerk muting and indefinite limbo.