Smoke Testing: The Check That Should Never Be Slow

Before spending forty minutes running a full suite, it is worth spending ninety seconds finding out whether the build is fundamentally broken. That is the entire job of a smoke test, and almost every problem teams have with them comes from forgetting it.

What a smoke test is

A smoke test is a small, fast set of checks that confirms a build is functional enough to be worth testing further. It is a gate, not an assessment.

The name comes from hardware: power on the board, and if smoke comes out, stop. Nobody runs a full diagnostic on a component that is on fire.

Applied to software, a smoke test answers one question. Does the application start, and do its most critical paths work at all? If the answer is no, the detailed suite would only produce a long list of failures caused by the same root problem, and the time spent producing that list is wasted.

What belongs in one

The test for inclusion is severity, not importance. Ask: if this were broken, would running anything else be pointless?

For a typical web application that usually means:

• The application loads
• A user can authenticate
• The main navigation works
• The core action of the product succeeds once, on a happy path
• Critical integrations respond

That is roughly it. Five to ten checks, finishing in a minute or two.

What does not belong: edge cases, error handling, validation messages, permutations, anything unusual. Those matter and they belong in the regression suite. Putting them in the smoke suite is the single most common mistake, because each one is individually defensible and collectively they destroy the thing that made the suite useful.

How it differs from neighbors

Smoke is broad and shallow: touch everything critical, examine nothing closely.

Sanity is narrow and focused: a quick check that one particular change did what it claimed, without running the world.

Regression is broad and deep, and it is what most people mean when they say "the test suite."

The distinction that matters operationally is what happens on failure. A smoke failure stops the pipeline. A regression failure means investigating one thing.

The failure mode: suite creep

Every smoke suite grows. The mechanism is always the same and it is worth naming because it is preventable.

A bug reaches production. In the retrospective someone asks how it escaped. Someone else proposes adding a check to the smoke suite, because that runs first and would have caught it earliest. It is a reasonable suggestion and nobody objects.

Repeat that fifteen times over a year, and the ninety-second gate takes twelve minutes. Developers start pushing without waiting for it. The gate that was valuable because it was fast is now a gate people route around, which is worse than not having one.

Two rules keep it honest:

A time budget, enforced. Pick a number, two minutes is a common one, and treat exceeding it as a defect. When something new must go in, something else comes out or gets faster.

The severity test, applied every time. "Would running anything else be pointless if this broke?" A validation message failing does not make the rest of the suite pointless. Login failing does.

Where it runs

Smoke tests earn the most in three places:

On every pull request, first. Fail fast before the expensive stages start.

Against a preview deployment, so you are checking the built artifact in a realistic environment rather than a developer's machine. See how to test Vercel preview deployments automatically.

Immediately after a production deploy. The highest-value smoke run of all, because it is the one that tells you to roll back. A deploy nobody verified is a deploy you find out about from a customer.

What changes when agents write the code

The economics shifted, and it cuts in a specific direction for this kind of test.

A smoke suite was historically hand-written and hand-maintained, so keeping it small was partly a resource constraint dressed up as discipline. When the agent that writes a feature can also exercise it in a real browser, generating checks stops being the bottleneck. See shift left testing.

That does not mean the smoke suite should grow. It means the regression suite can grow, absorbing the checks that used to get pushed into smoke for lack of anywhere better. The gate stays small because there is now a good home for everything else.

The other change is what needs smoke-testing. Agents refactor aggressively, so the "did the app fundamentally survive that" question gets asked more often per day than it used to. A characteristic agent failure is a quietly removed safeguard rather than an obvious break, which smoke will not catch. Smoke is a viability gate, not a safety net. See regression risk in AI-generated code.

Shiplight fits this by letting the agent verify a change in a real browser and commit the check as intent-based YAML, so smoke and regression suites both grow without the maintenance cost that used to cap them. Tests are tagged and run in whichever stage you choose. Web applications; native mobile is not in it.

Frequently Asked Questions

Related Reading

• End-to-end testing: the fuller suite smoke sits in front of
• Shift left testing: why generating checks got cheap
• How to test Vercel preview deployments: where smoke runs best
• E2E testing in GitHub Actions: wiring the gate into CI