End-to-end testing

One shop’s end-to-end suite on one screen — twelve journeys, the seam each proves, and what is real

Nobody opens a page about end-to-end testing for the definition. They open it because a suite of four hundred browser tests takes an hour and fails every other run, or because someone asked how many such tests a team should have. Below is the whole end-to-end suite of the online shop whose smoke gate and regression suite the two neighbouring pages list. It has twelve journeys. Each row names the seam only that journey proves — a place where two real parts of the system meet and no lower test can reach — and says what is real and what is replaced. Later sections cite the journeys by number.

What only an end-to-end test proves

The ISTQBISTQBInternational Software Testing Qualifications Board — testing certification bodyThe international body behind the most widely taken software-testing certifications, which also publishes a glossary of testing terms. Its definitions are a common reference in interviews and contracts; they fix vocabulary, not how a team has to work.All terms glossary defines end-to-end testing as “a test type in which business processes are tested from start to finish under production-like circumstances”. Two phrases in it carry the weight. “From start to finish” means the journey ends at the outcome a customer needs — the parcel in the warehouse, the money back on the card — not at the last screen. “Production-like” means the configuration, the routing and the topology of the real thing. Notice too that ISTQB calls it a test type, not a level. The test pyramid draws it as the top layer, but what defines it is what it covers, not which part of the code it sits in.

Every test below it works in a controlled setting. A unit test replaces its neighbours with fakes. An integration test uses a real database and a stand-in for the payment provider. A contract test proves two services agree about a message without either running the other. All of that is valuable, and none of it can show that the deployed system, with its real routing, real environment variables, real session handling and real front-end build, lets a person finish what they came to do. The classic defect that escapes a well-tested system is not a wrong calculation. It is a misconfigured redirect, a cookiecookiedata the browser sends backA small piece of data a website asks the browser to keep and send back with every request to that site — which is how a server remembers that you are signed in. Because the browser attaches it automatically, even to a request another site triggered, a session cookie needs its flags set and an anti-forgery check on the server.All terms attribute that breaks the session in one browser, a feature flagfeature flaga switch around new codeA switch that decides at run time whether a piece of code is active, so it can ship to production turned off and be turned on for a chosen audience later. It separates deploying from releasing, and it costs something: every live flag is a branch in the code that has to work in both states.Full article →All terms on in stagingstagingthe rehearsal environmentA copy of production that changes are tried on before real users meet them. How much it proves depends entirely on how closely it matches production — a staging environment with a tenth of the data, no real traffic and a different config demonstrates that the code starts, and not much more.All terms and off in production. Each lives at a seam, and every lower test replaced exactly that seam.

The diagram is the mechanism behind every case at the end of this page. Split a journey into segments, test each segment with a stand-in at its joint, and every segment can be green while the journey fails — because the defect lives in the joint, and the joint is the one thing no segment test ran. Martin Fowler adds a warning about the words: teams “conflate the concepts of end-to-end tests, UIUIUser Interface — what the user seesEverything the user sees and operates: screens, buttons, fields, states. UI is how the interaction looks; whether the interaction makes sense in the first place is a UX question, and the two get confused constantly.All terms tests, and customer facing tests. These are all orthogonal characteristics.” A test that clicks through a browser against a mocked back end is a UI test. It is not end to end, and it cannot prove a seam it has replaced.

Pyramid, trophy, honeycomb — three shapes, one argument about cost

Mike Cohn described the test pyramid in his 2009 book Succeeding with Agile, and Martin Fowler’s 2012 note made it famous: many unit tests at the bottom, fewer service tests in the middle, few end-to-end tests at the top, because tests that run end to end through the interface are “brittle, expensive to write, and time consuming to run”. ISTQB’s glossary now has an entry for it. Since then other shapes have been drawn, each by people working on a different kind of system, and the table puts them side by side.

The shapes disagree about the middle and agree about the top. Spotify’s honeycomb wants “even fewer Integrated Tests (ideally none)”, where an integrated test is one “that will pass or fail based on the correctness of another system”. Google’s 2015 post settles for about a tenth. Dodds keeps a few. The disagreement is real and worth leaving visible: a team of independent services can test most interactions through contracts and each service’s APIAPIApplication Programming Interface — contract between programsThe rules one program follows to call another: which requests it may send, in what shape, and what comes back. To the caller it is a contract — everything behind it stays a black box, which is exactly what lets the other side be rewritten without breaking you.Full article →All terms, while a shop that takes money across five systems in one customer’s minute cannot. What none of them says is that the top layer should be large. Fowler states the reason it is small as an assumption rather than a law: “If my high level tests are fast, reliable, and cheap to modify - then lower-level tests aren’t needed.” They rarely are.

What a red journey costs — the arithmetic of flakiness

The runtime is the obvious cost and the least important one. The real expense is the distance between a failure and its cause. When a unit test fails you are looking at one function. When journey 9 fails you are looking at a browser, a front end, a warehouse webhookwebhookan HTTP call saying something happenedAn HTTP request one system sends to a URL you registered, to tell you that something happened — a payment succeeded, an order shipped — so you do not have to keep asking. Delivery is usually at least once and sometimes out of order, so the receiver must cope with the same event arriving twice.Full article →All terms, a refund service, a payment sandbox, an email provider and an environment — and possibly at none of them, because the test may simply have checked the inbox before the email arrived.

Size makes that likelier. Google found that over a week 0.5% of its small tests were flakyflakypasses and fails on the same codeA test that passes and fails on unchanged code. The real damage is not the failed run but what follows: once a suite has a few, people start re-running until green, and the suite stops being evidence of anything.Full article →All terms, 1.6% of its medium tests and 14% of its large ones. A journey is the largest test a team owns. And small rates compound: if each case fails for no reason with probability p, a run of n cases goes red with nothing wrong with probability 1 − (1 − p)ⁿ. The table works it out.

Read the table as a budget, not a complaint. A suite of sixty journeys at 2% each is red on seven runs out of ten when nothing is wrong. Four hundred journeys at the best rate in the table are red almost nine times in ten. That is why “we will just add retries” is the wrong first move, and why ten times the journeys is not ten times the protection — it is a suite that will be ignored. Twelve journeys at 0.5% go red for no reason about once in seventeen runs, which a team can afford to investigate every time.

Choose journeys, not features

The selection mistake almost every team makes is one end-to-end test per feature. Features are how the backlog is organised. They are not how a customer moves through the product, and covering them one by one gives hundreds of slow cases that mostly re-prove logic already proved below. Choose journeys instead: a small number of paths a customer actually completes, each crossing several services, because the crossing is what is being tested.

• The money path — journeys 1, 4–6

  Find, add, pay, and the order arrives. If exactly one end-to-end test exists, it is journey 1, because it crosses every service the business depends on.

• Getting in and staying in — journeys 2, 3, 11

  Sign up, sign in, stay signed in after a reload, recover a password. Session and cookie behaviour is invisible to every lower test.

• Across a boundary — journeys 7–10

  Work one service starts and another finishes, often hours later in real life: a return the warehouse confirms, a refund an email reports.

• A hard rule — journey 12

  A permission boundary, a regulated disclosure, an age check. Places where being wrong is a legal event rather than a support ticket.

A sizing heuristic, the one in the callout at the top: one journey for each thing the business could not survive losing, plus one for each seam that exists only in the deployed system. For most products that lands between ten and forty. If yours is in the hundreds, the suite is not more thorough than everyone else’s; it is carrying work that belongs two levels down and paying the highest price per case in the codebase to do it. And when a journey finds a bug, follow Fowler’s rule from the Regression testing page: reproduce it at the lowest level that can, so the journey is the second line of defence, not the only one.

Making twelve journeys trustworthy

A trustworthy end-to-end suite does not come from better assertions. It comes from removing every source of variation that is not the behaviour under test, and five of them are worth handling on purpose.

• Wait for a condition, never for a duration. A fixed sleep is too short on a loaded machine and wasted on a fast one; waiting for the element, the email or the warehouse record is both faster and stable.
• Give every journey its own data. It creates the customer, the order and the gift card it needs, and removes them afterwards. A journey that depends on a record someone else made fails on a Tuesday for no reason.
• Find elements the way a person does — by role and visible name — or by a test-idtest-idan attribute tests hold on toAn attribute added to markup for the sole purpose of letting tests find an element. It exists so a test does not hold on to a CSS class or a piece of visible text — both of which change for reasons that have nothing to do with behaviour.Full article →All terms, never by a CSS class or a position, both of which change whenever a designer touches the page.
• Control the clock. A journey that depends on today’s date passes in March and fails on the last day of a month; set the date, and then walk the dates that break things on purpose — the 30th day of a return, the 29th of February.
• Capture enough on failure to diagnose without reproducing: a trace, a video, the network log and the browser console. The cost of a red journey is the investigation, and artefacts are what shorten it.

The sixth source is the outside world, and it needs a decision per dependency rather than a default. The test asks one question about each: is this part of the journey I am proving, or something the journey merely touches?

Retries deserve a policy rather than a default. Retrying a whole journey hides real intermittent bugs, because a race in the product looks exactly like a flaky test from the outside, and the retry turns a genuine defect into a green tick. The shop’s rule is the one on the Regression testing page: one retry at most, recorded as a signal in its own right, and a journey that needed two retries in a week goes to quarantine. A suite whose green result depends on retries is not green; it is unmeasured.

Three failures where a test stopped one step short

In none of these was the testing skipped. Each time a test ran and passed. It checked a step short of the outcome that mattered, stopped just before the seam where the defect lived, or ran in a place that was not shaped like production.

Where it degrades

The honest limit is that twelve journeys prove twelve paths worked once, in one environment, with one set of data. Production-like is not production: the real traffic, the real third parties and the real calendar arrive only after the deploy. That is where the smoke gate and synthetic monitoring take over, running the same journeys against production on a schedule. And an end-to-end suite cannot find a defect on a path nobody walked; it can only keep the walked paths from breaking unnoticed.

• An ice-cream cone: most of the testing done through the browser, few tests below, so every failure is an hour of investigation.
• One test per feature, so four hundred cases mostly re-prove logic already proved in milliseconds.
• A browser test against a mocked back end, reported as end to end.
• Journeys that end on the confirmation screen instead of at the outcome — the parcel, the refund, the email.
• One long case walking fifteen screens, so a failure says only that something in fifteen screens is broken.
• Journeys that depend on running in order, so one failure cascades into six and the report describes nothing.
• A suite run only at night, so a regression is blamed on a day of merges rather than on one commit.
• An environment kept by hand that nobody can rebuild, which fails for its own reasons more often than for the code’s.
• Retries on everything, and sleeps scattered through the code as the standard fix.

When to use it

Related pages

• QASmoke testingSmoke testing is a small, fixed set of checks run right after a build is deployed, deciding whether that build is worth anyone else’s time. The page opens with one shop’s finished suite — eleven checks, what each proves and what a red result stops — then covers where the term comes from, what belongs in the suite, how it differs from health checks and monitoring, where it runs, the names people confuse, and four deployments where the check was missing or ignored.
• QARegression testingRegression testing is re-running tests after a change to catch anything that used to work and has stopped, in parts of the system nobody meant to touch. The page opens with sixteen cases from one shop’s suite — why each is there, the level that proves it and when it runs — then covers what earns a case and what removes one, which subset runs when, flaky tests by the numbers, the cost nobody budgets for, and four regressions where the case was missing.
• QAPlaywrightPlaywright is an open-source browser automation library and test runner from Microsoft that drives Chromium, Firefox and WebKit from outside the browser. The page opens with the twelve journeys from End-to-end testing written in Playwright — the mechanism each one leans on, and the config that runs them in about four minutes — then covers the one connection everything follows from, the browsers it actually runs, auto-waiting, contexts and workers, what a test can reach besides the page, traces and retries, four changes underneath working suites, and where it does not help.