Regression testing

One shop’s regression suite on one screen — sixteen cases, why each is there, and when it runs

Nobody opens a page about regression testing for the definition. They open it because a bug came back, or because the suite now takes forty minutes and people merge without waiting. Below are sixteen cases from the suite of the online shop whose smoke gate the Smoke testingsmoke testis this build worth testing?A small, fixed set of checks run right after a build is deployed, answering one question: is this build worth anyone else’s time? It is a gate, not coverage — it proves the build is testable, never that it is correct.Full article →All terms page lists. Rows 1–4 came from bugs that reached customers, 5–8 from incident reviews, 9–12 from promises the shop has made, and 13–16 from the paths money takes. Each row names the lowest level that can prove it and the tier it runs in. Later sections cite the rows by number.

A regression is a broken promise in a place nobody meant to touch

The ISTQBISTQBInternational Software Testing Qualifications Board — testing certification bodyThe international body behind the most widely taken software-testing certifications, which also publishes a glossary of testing terms. Its definitions are a common reference in interviews and contracts; they fix vocabulary, not how a team has to work.All terms glossary defines regression testing as change-related testing “to detect whether defects have been introduced or uncovered in unchanged areas of the software”. Two words in that sentence carry the meaning. “Unchanged” says where to look: not at the feature you edited, which its own tests already cover, but at everything around it. And “uncovered” admits something teams forget. A change can expose a defect that was already there, sitting quietly behind behaviour that has now moved.

Row 1 carries the whole mechanism. A developer changed how invoices round, and did it correctly. The loyalty points calculation used the same helper, and nobody editing invoices had reason to know that. Regressions come from coupling you cannot see from the code you are editing: a shared function, a shared table, a shared cache, a configuration value two features read for different reasons, an ordering nobody wrote down. Nobody was careless. The reach of a change is simply not visible from inside it, and it grows with every year the system stays alive.

The diagram is the reason this page is sceptical of clever test selection. A change reaches other features along two kinds of path. Along code paths — one function calling another — a tool can follow it and pick the tests that run that code. Along data and configuration paths — one feature writing a table that a monthly job reads later — no code connects the two, so nothing selects the job’s tests. Those are exactly the paths regressions take most often, which is why the full tier exists.

What earns a permanent case — and the one thing that removes it

A suite without an entry rule grows by taste, and taste has no upper bound. The rule that works is narrow: a case earns permanence when the behaviour has already shown that it matters. Four things show that, and they are the four groups of the table at the top. “Someone thought it would be thorough” is not among them.

• A bug that reached a customer — rows 1–4

  The strongest source there is. It has been shown that this can break, that nothing caught it, and that someone outside the team paid for it.

• An incident review — rows 5–8

  The review names a condition nobody was checking. Turning that sentence into a case is the part of a review that survives the quarter.

• A promise — rows 9–12

  A published API shape, a returns policy, a regulator’s rule. The promise exists whether or not a test does, so make the test exist.

• A path money takes — rows 13–16

  Checkout, refunds, gift cards. Failures here are not reported by customers filing tickets; they are found by an accountant three weeks later.

Where a case lives matters as much as whether it exists. Martin Fowler’s advice on the test pyramidtest pyramidmore small tests than big onesA model of a test suite with many fast, narrow tests at the bottom, fewer that test services in the middle, and few end-to-end tests at the top. It is an argument about cost, not a quota: the top layer is small because each test there is the slowest and most expensive one a team owns.Full article →All terms is to replicate a bug exposed by a high-level test with a unit test before fixing it: “Then the unit test ensures the bug stays dead.” The shop does exactly that. Row 1 was reported by customers who looked at their points, yet its case is a unit test that runs in milliseconds and names its own cause when it fails. Row 3 is a browser journey only because a Safari cookiecookiedata the browser sends backA small piece of data a website asks the browser to keep and send back with every request to that site — which is how a server remembers that you are signed in. Because the browser attaches it automatically, even to a request another site triggered, a session cookie needs its flags set and an anti-forgery check on the server.All terms cannot be proved anywhere below a real Safari.

The exit rule is stricter, and it is the one teams get wrong. A case leaves the suite when the behaviour leaves the product — the feature was removed, the rule was deliberately changed, the endpoint was retired. Every other reason to delete a case has a better answer, and the table lists the usual ones.

What runs when — the suite outgrows the wait

Every regression suite eventually crosses a line. It gains cases every release, loses them almost never, and one day running all of it on every change takes longer than a developer will wait. People then stop waiting: they merge, start the next task, and read the result after it no longer matters. From that day the interesting question is not “what should we test” but “which subset runs at which moment”, and that is a design decision about risk and feedback speed.

A useful default: parallelise first, tier second, and be sceptical of clever selection until the number of changes a day makes it unavoidable. Buying machines is cheaper than an engineer working out why a regression slipped through the smart subset. Whatever you choose, two rules are not negotiable. The merge tier tests each change merged with the current main branch, which is what a merge queue does — the Git branching page explains why two green pull requestspull requesta change proposed for reviewA request to merge one branch into another, where the change is reviewed, discussed and checked by the pipeline before it reaches the main branch; GitLab calls the same thing a merge request. Its cost is waiting: a pull request that sits unreviewed for days is where long-lived branches and painful merges come from.Full article →All terms can still break main together. And the full suite runs before every release, because that run is the last moment the memory is complete.

Flaky tests — measured, not mysterious

A flaky test passes and fails on the same code. Google published its numbers in 2016: about 1.5% of all test runs reported a flaky result, almost 16% of tests had some level of flakiness, and about 84% of the transitions from pass to fail that its CICI/CDContinuous Integration / Continuous Delivery — automatic build, test and deliveryEvery commit is built and put through the tests automatically (CI), and a build that passes travels to the servers the same way (CD). The point is not automation for its own sake: it is that a breakage shows up in minutes instead of on release day.Full article →All terms system observed involved a flaky test. Read the last figure twice. Five times out of six, the test that has just gone red is one that also fails with no change at all. On an average project of about a thousand tests, 1.5% means some fifteen failures on every release that somebody has to investigate.

The damage is not the wasted investigation. It is what people learn from it. Martin Fowler put it plainly in 2011: once you start ignoring a regression test failure, that test is useless. Worse, the habit spreads: a team that shrugs at the flaky cases soon shrugs at the healthy ones too. Size makes it worse. A year after the first post, Google found that 0.5% of its small tests, 1.6% of medium tests and 14% of large tests had been flaky in a week, and that size predicted flakiness better than the tool a test was written with.

The causes are known, too. Luo and colleagues studied 201 commits that fixed flaky tests in 51 open-source projects and classified the cause of 161. Three causes covered more than three quarters of them, and each has a fix that removes it rather than hiding it.

Two more findings from the same study change how a team should treat a flaky case. 78% of the tests were flaky from the day they were written, so a new case that runs fifty times before it may block anything gives most flaky cases a chance to show themselves before anyone learns to ignore them. And 24% of the fixes changed the code under test, not the test. A quarter of the time, the “flaky test” was a real defect that a test had found and everyone had decided to call flaky.

Test data, upkeep, and the numbers worth watching

A regression suite has a running cost that arrives whether or not anyone planned it. Cases break when the interface is redesigned, when an API adds a field, when test data drifts, when a date rolls over a year boundary. Budget for that upkeep explicitly — a share of every sprint — because the alternative is not a cheaper suite. The alternative is a suite that decays until people ignore it, which costs the same and protects nothing.

The suite is a product with users, and it deserves the same few numbers a product gets. Case count and code coverage are not among them: both grow when cases are added, whether or not the cases protect anything. The table lists the ones that say whether the memory works.

Four regressions, and the case that was missing

None of these organisations was short of tests. In each, a change that looked safe — a repair, a routine data update, an edit, a fix for an earlier complaint — broke something that had worked, and in each the case that would have caught it can be named in one sentence.

Where it degrades

The honest limit is in the name. A regression suite remembers; it does not discover. Every case checks something somebody already thought of, so the suite finds fewer new defects every year it runs unchanged, and a green run says only that none of the remembered things broke. New risk needs new cases — from reviews, from exploratory testing, from production signals — and the suite is where they go afterwards, not where they come from.

It degrades into theatre in recognisable ways, and most of them are ways of keeping the green colour while losing the memory.

• A suite that only grows, with no exit rule, so a third of it tests behaviour the product no longer has.
• Regression run as a phase before release rather than on every change, so defects are found at the most expensive moment.
• Every case written as a browser test because that is what the QA team owns, where an API test would prove the same in a hundredth of the time.
• A quarantine list with no owner and no limit — a graveyard with a more optimistic name.
• Automatic retries everywhere, so the pipeline is green and nobody knows how often it was red first.
• The full tier skipped “just this once” before a release, which is the one run whose absence nobody notices until a customer does.
• Measuring the suite by case count or code coverage rather than by regressions that reached customers.
• A fix closed when the reported steps pass, with no case left behind — the bug is gone, and so is the memory of it.

When to use it

Related pages

• QASmoke testingSmoke testing is a small, fixed set of checks run right after a build is deployed, deciding whether that build is worth anyone else’s time. The page opens with one shop’s finished suite — eleven checks, what each proves and what a red result stops — then covers where the term comes from, what belongs in the suite, how it differs from health checks and monitoring, where it runs, the names people confuse, and four deployments where the check was missing or ignored.
• QAEnd-to-end testingEnd-to-end testing walks a business process through the assembled system from start to finish, the way a customer would, in conditions close to production. The page opens with one shop’s whole suite — twelve journeys, the seam only each one proves, and what is real and what is replaced — then covers what no lower level can prove, the shapes a portfolio takes, what a red journey costs, how to choose journeys and keep them trustworthy, and three failures where a test stopped one step short.
• ManagementGit Branching StrategyA Git branching strategy is the agreement on how a team splits its work into branches and rejoins them. The choice looks like a matter of taste and is not: the variable that predicts pain is how long a branch lives before it merges, and every model is a position on that number plus a rule for where releases and fixes come from. The page opens with six models on one screen and the five questions that choose between them, then covers branch lifetime, where each model came from and what its authors said later, release branches and the way a fix travels, what makes short branches possible, merge versus rebase versus squash, and where it degrades.