Regression testing
Regression testing is the practice of quickly catching a regression — something that used to work stopping working, when nobody meant to touch it. The suite it produces is best understood as institutional memory: a record of everything the system has already been taught, kept in a form that can be replayed. The hard parts are not writing the cases. They are deciding what earns a permanent place, what runs when the whole suite is too slow to run, and what to do about a case that fails for no reason.
- It answers
- Did we break the past?
- Entry rule
- It happened once
- Exit rule
- The behaviour is gone
A regression is a broken promise you did not know you had made
Every piece of working software is a set of promises. Some are written down as requirements, some live in a contract with another team, and a great many exist only because a user built a habit on top of them. A regression is what happens when a change breaks one of those promises without anyone intending to — the developer changed a rounding rule for invoices and the loyalty points calculation, which quietly used the same helper, started paying out three cents too little.
That example carries the whole mechanism. Regressions come from coupling you cannot see from the code you are editing: a shared function, a shared table, a shared cache, an implicit ordering, a configuration value that two features read for different reasons. Nobody is careless. The blast radius of a change is simply not visible from inside the change, and it grows with every year the system stays alive.
What earns a permanent case — and what earns removal
A suite without an entry rule grows by taste, and taste has no upper bound. The rule that works is narrow: a case earns permanence when the behaviour has already proved it matters. Four things prove that, and "someone thought it would be thorough" is not among them.
A bug that reached a user
The strongest source there is. It has been demonstrated that this can break, that nothing caught it, and that someone outside the team paid for it.
An incident post-mortem
The review names a condition nobody was checking. Turning that sentence into a case is the only part of a post-mortem that survives the quarter.
A behaviour you promised
A published API response shape, a contractual SLA, a regulatory rule. The promise exists whether or not a test does, so make the test exist.
A path money flows through
Checkout, billing, payouts, refunds. Failures here are not caught by users filing tickets; they are caught by an accountant three weeks later.
The exit rule is stricter and it is the one teams get wrong. A case leaves the suite when the behaviour leaves the product — the feature was removed, the rule was deliberately changed, the endpoint was retired. It does not leave because it is slow, because it is annoying, or because it failed this morning and the release is today. Deleting a case for those reasons deletes the memory of the bug it was written for, and that bug is now free to come back with nothing watching.
The real problem: the suite is too slow to run in full
Every regression suite eventually crosses a line. It grows every release, loses cases almost never, and one day running all of it on every change takes forty minutes — which is longer than a developer will wait, so people stop waiting. At that point the interesting question is no longer "what should we test" but "which subset runs at which moment", and that is a design decision about risk and feedback speed rather than a line somebody happened to write in a CI file.
| Selection strategy | How it decides | What it costs you |
|---|---|---|
| Run everything, every time | No decision at all — the whole suite on every change. | Works beautifully until it does not. When the wait exceeds patience, people start merging without reading the result. |
| Tiers by cadence | Fast tier on every commit, middle tier on every merge, everything nightly and before release. | A regression can hide for a day. The default answer for most teams, and the easiest to explain. |
| Risk-based selection | A human decides which areas a change threatens and runs those, based on how the system is actually wired. | Only as good as the person’s model of the system, and it silently degrades as they forget or leave. |
| Test-impact analysis | Tooling maps which tests execute which code, then runs only the tests touching the changed lines. | Real speed-ups on big codebases, but blind to coupling through data, configuration and time. |
| Parallelise instead of selecting | Shard the suite across many machines so the whole thing finishes in the time of its slowest shard. | Buys years without weakening coverage, and costs money and isolated test data instead. |
A useful default: parallelise first, tier second, and be sceptical of clever selection. Buying machines is cheap compared with an engineer debugging why a regression escaped the smart subset, and every selection scheme is blind in the same place — coupling through shared data and shared configuration, which is exactly where regressions come from. Whatever you choose, one rule is not negotiable: the full suite runs before a release, no exceptions, because that run is the last moment the memory is complete.
Flakiness, test data, and the cost nobody budgets for
A regression suite has a running cost that arrives whether or not anyone planned it. Cases break when the UI is redesigned, when an API adds a field, when a fixture drifts, when a date rolls over a year boundary. Budget for that maintenance explicitly — a share of every sprint — because the alternative is not a cheaper suite. The alternative is a suite that decays until people ignore it, which costs the same and protects nothing.
Quarantine — the right move
- The case is removed from the blocking gate the same day it is confirmed flaky.
- It keeps running and keeps reporting, into a separate list with a named owner.
- A deadline is attached: fixed within two weeks, or the behaviour is re-covered another way.
- The suite stays believable, which is the only property that makes a red result mean anything.
Re-run until green — the trap
- Costs nothing today, which is precisely why it becomes the default within a month.
- Trains everyone that a red result means "press the button again" rather than "stop".
- Hides real intermittent bugs — race conditions look exactly like flaky tests from the outside.
- Ends with a suite nobody reads and a release process that relies on hope.
Most flakiness has one of four causes, and naming them turns a mysterious problem into a work item. Timing — the test asserts before the application has finished, which is fixed by waiting for a condition rather than for a duration. Shared state — two cases use the same account or the same row, so they pass alone and fail together. Environment — a slow network, a cold cache, a neighbour saturating the machine. And genuine non-determinism in the code itself, which is not a test problem at all: it is a bug the test found and everyone decided to call flaky.
Anti-patterns worth naming
- A suite that only ever grows, with no exit rule, so a third of it tests behaviour the product no longer has.
- Regression testing scheduled as a phase before release rather than run continuously — the defects are then found at the worst possible moment.
- Every regression case written as a browser test, because that is what QA owns, even where an API test would prove the same thing in a hundredth of the time.
- A quarantine list with no owner and no deadline, which is a graveyard with a more optimistic name.
- Measuring the suite by case count or code coverage rather than by defects that escaped to users, which is the only number that describes whether it works.
When to use it
Use it when
- Any system that will be changed again, which is every system that is not being switched off this month.
- Immediately after a bug is fixed: confirm the fix, then keep the case permanently so the bug cannot return unseen.
- Around anything with a promise attached — a published API, a contractual SLA, a regulatory calculation.
- Split into tiers by cadence once the full suite is slower than the wait people will actually tolerate.
Avoid it when
- As a phase performed only before a release — a regression found then is found at the most expensive moment.
- Adding a case for every scenario anyone can imagine, which produces a suite too slow to run and too big to maintain.
- At the browser level for logic an API or unit test could prove, where the same coverage costs a hundred times more to run and to fix.
- With flaky cases left inside the blocking gate, because the first ignored red result is the moment the suite stops working.
Found this useful?
Share it with someone who is working on the same problem.