Product Discovery

A quarter of discovery on one screen — finished

Most people who look up product discovery want to see what a team actually does, and what comes out of it. Here is one quarter of it, for the customer team of the online shop used across the Decomposition and Release Process pages. It is the quarter that produced the reorder epicepica large backlog itemA large piece of work that is split into features or stories before anyone builds it, best written as a bet with a hypothesis and a condition that closes it. Its size depends on whose vocabulary is in use — anything above a story in Jira, a portfolio investment with a business case in SAFe — so check before comparing two epics.Full article →All terms and the key result it serves. Later sections cite the lines by number.

1. 1.The outcome, the team, the rhythm

   1. 1.1Outcome: week-2 return visits from 18% to 25% — a product outcome, later written down as key result 1.1 on the OKROKRObjectives and Key Results — a goal with measured resultsA goal-setting method that pairs one qualitative objective with two to five measurable key results proving it was reached, set and reviewed every quarter. It stops working when the key results become a task list or the scores feed into pay.Full article →All terms page
   2. 1.2The trio: the product manager, the designer and the tech lead, all three in every interview
   3. 1.3Two interviews a week, Tuesday and Thursday at 10:00. Customers are recruited by a prompt shown after their third order, with a voucher for taking part
   4. 1.4The tree is redrawn every four weeks, after eight interviews

2. 2.What sixteen interviews and the order data showed — in customers’ words

   1. 2.1“I buy the same things every two weeks, and finding each one again takes ages” — 11 of 16 interviews. In the order data, 58% of repeat orders contain at least 70% of an earlier basket
   2. 2.2“I only remember to order when something has run out” — 7 of 16
   3. 2.3“For one item, the delivery costs more than the item” — 6 of 16
   4. 2.4“I’m never sure the price is the same as last time” — 5 of 16
   5. 2.5“The payment failed and I don’t know whether I was charged” — 4 of 16

3. 3.The target opportunity, and the solutions compared

   1. 3.1Target: 2.1 — the most frequent, and the only one the order data confirms on its own
   2. 3.2A — Reorder from order history: a past order goes into the cart
   3. 3.3B — Saved baskets: a named list the customer keeps
   4. 3.4C — One-tap buy: a past order placed and paid at once
   5. 3.5D — Subscriptions, delivered every few weeks — parked as a bet of its own

4. 4.The assumptions under A — how much it needs each, and what is known

   1. 4.1Value: returning customers repeat most of an earlier basket — known, from 2.1
   2. 4.2Value: they will look for it in their order history rather than in search — not known
   3. 4.3Usability: they will notice what is out of stock and what changed price — not known
   4. 4.4Feasibility: the ERP can report stock per item in real time — not known; the tech lead’s question
   5. 4.5Viability: an order is confirmed on a button that says it means paying, with the total shown just before — known, because the law requires it, and it rules C out

5. 5.Tests — each success criterion written before the test ran

   1. 5.1Order data, two hours, for 4.1: criterion — at least 40% of repeat orders repeat an earlier basket; result 58%
   2. 5.2Fake door, one week, 5% of returning customers, for 4.2: a Reorder button in order history that opens a “coming soon” note. Criterion — at least 8% of history visitors tap it; result 14%
   3. 5.3Prototype, five customers, for 4.3: criterion — four of five reorder and spot the missing item; result four of five, but two of five missed a changed price
   4. 5.4SpikeSpiketime-boxed researchA deliberately time-boxed piece of research whose output is an answer, not a feature. The time box is the whole point: a spike without one becomes an open-ended investigation that quietly consumes a sprint.Full article →All terms, two days, for 4.4: written into the epic as its line 4.3, to be answered before feature 1 is built

6. 6.Decisions

   1. 6.1Build A as feature 1 of the reorder epic, with story 1.4 — a changed price shown before payment — added because of 5.3
   2. 6.2B becomes feature 2. Reminders, for 2.2, become feature 3, built only if the leading indicators move
   3. 6.3C is dropped: it fails 4.5, and an order placed by mistake cannot be undone by switching a flag off — line 4.3 of the release plan
   4. 6.42.3 goes to the pricing team; 2.5 goes to objective 2 on the OKR page, payments
   5. 6.5The outcome stays for the next quarter. Discovery on 2.1 stops when the epic’s closing condition is met

Discovery beside delivery, not before it

Teresa Torres defines the work by contrast: discovery is what a team does to decide what to build, and delivery is what it does to build, ship and maintain it. Her shortest version is that discovery is “a decision-making process”. The traditional shape puts a discovery phase first, producing requirements, and a delivery phase after it. It fails for a specific reason: the most valuable thing you learn about a product is what happens when people use it, and that information only exists after delivery has started.

So the two run side by side. Desirée Sy described one way of doing it in May 2007, from Alias’s design team: designers worked a cycle ahead of the developers, testing the next cycle’s designs while the current one was being built. Jeff Patton called the pattern dual-track, and Marty Cagan popularised it in September 2012 with a definition worth keeping: “The Discovery track is all about quickly generating validated product backlog items, and the Delivery track is all about generating releasable software.”

Why it has to be three people

Torres’s product trio is a product manager, a designer and a software engineer, and line 1.2 puts all three in every interview. This is not a coordination convenience. Each of them hears something the others miss: the product manager hears the business consequence, the designer hears where the customer’s mental model breaks, and the engineer hears which constraint just became irrelevant or just became fatal. Send one of them and the other two get a summary — a different artefact, because most of what is useful in a conversation is the thing nobody knew to write down.

The engineer is the one most often left out, and the most expensive to lose. An engineer who heard the problem proposes cheaper ways to solve it, including some with no interface at all, before anyone has fallen in love with a design. In the shop’s quarter it was the tech lead who raised 4.4, the one question that could have stopped solution A in the second sprint. An engineer who receives the solution afterwards can only estimate it.

The opportunity solution tree

Torres introduced the opportunity solution tree in 2016, and it is the structure that keeps discovery honest. It has four levels. At the root is one desired outcome. Below it are the opportunities: “the customer needs, pain points, and desires that, if addressed, will drive your desired outcome”. Below each opportunity are solutions, and below each solution are the tests of its assumptions. A solution is therefore always compared with other solutions for the same need rather than judged alone.

The root has to be the right kind of number. Torres separates three. A business outcome, such as revenue, matters to the company but is moved by too many things for one team to steer. A traction metric, such as “uses of the Reorder button”, is too narrow, because it already names the solution. A product outcome sits in between: it measures customer behaviour the team can influence, and it leaves the solution open. “I like to see a product outcome at the top,” she writes. Line 1.1 is one: week-2 return visits, which any of solutions A to D could move.

An opportunity is stated in the customer’s terms — “finding each one again takes ages”, not “add a reorder button”. Keeping that level free of solutions is what makes the tree work. The moment an opportunity is phrased as a feature, every alternative underneath it can only be a variation of that feature, and nothing is really compared. Torres’s advice on the rest is about pace: choose a target opportunity “as soon as you have a first draft”, then “choose three solutions to explore further”. Group 3 has four, and one of them, C, lost on a single assumption.

Interviews — stories about the past, not opinions about the future

The question that makes an interview useful is not “what would you like?” but “tell me about the last time you did this”. The first invites the customer to design, which people do badly and confidently, using only solutions they have already seen. The second produces a specific story with dates, workarounds and things that went wrong. That is evidence rather than opinion, and it can be checked against the next person’s story. Line 2.1 is what eleven such stories had in common.

Rob Fitzpatrick’s “The Mom Test” (2013) is named after the problem: even your mother will lie to you about your idea, out of kindness. Its three rules fit on a card. Talk about their life instead of your idea. Ask about specifics in the past instead of generics or opinions about the future. Talk less and listen more.

Two habits turn interviews into a tree. After each interview the trio writes a one-page snapshot — who the customer is, the story they told, the opportunities in it, in their words — which Torres calls an interview snapshot. And opportunities are counted across interviews rather than remembered: “11 of 16” in line 2.1 is a count, and a count survives the most recent vivid interview, which memory does not.

Four risks, and the assumptions under a solution

Cagan’s four big risks, as he set them out in December 2017, are the questions a solution must survive before engineering time is spent on it. Teams reliably test feasibility and skip value, which is the wrong order: feasibility is the risk engineers can see, and value is the one that kills products. His advice is to “tackle the big risks early — especially value risk and business risk”.

Line 4.5 is the case that usually surfaces too late. Under EU consumer law, an online order that commits the customer to pay must be placed on a button “labelled … only with the words ‘order with obligation to pay’ or a corresponding unambiguous formulation”, with the total shown directly before it. If it is not, the customer “shall not be bound by the contract or order”. That is Directive 2011/83/EU, article 8(2). One-tap buy from an order list fails it by design, and one question to the legal team settled what a month of building would otherwise have discovered.

From idea to test: assumption mapping

A solution is rarely wrong as a whole. It is wrong because one belief underneath it does not hold. Assumption mapping, which David Bland and Alexander Osterwalder set out in “Testing Business Ideas” (2019), is the step between “we could build this” and “here is what we will test”. The trio writes down every belief the solution depends on, then places each on two axes: how much the idea depends on it — if it is false, does the idea die? — and how much evidence already exists.

Only the important assumptions with no evidence are worth a test: 4.2, 4.3 and 4.4. The important ones with evidence, 4.1 and 4.5, are safe to build on. The shop’s trio wrote down eleven beliefs, and six of them landed in the bottom half — true or not, the idea would survive. Sorting them took about twenty minutes and a wall, and it is why group 5 has four tests rather than eleven.

Tests, from cheapest to dearest

Tests differ in what they cost and in what kind of evidence they produce. The difference that matters most is between what people say and what people do. A customer who says they would use something has spent nothing. A customer who taps a button, joins a waiting list or pays has spent something, and that is much stronger evidence.

The bottom rung is dearer than it looks. To tell week-2 return visits of 18% from 19% with the usual confidence — a 5% false-positive rate and an 80% chance of catching a real effect — an A/B test needs about 24,000 customers in each group. For 18% against 20% it needs about 6,000. A shop with 3,000 returning customers a week would wait sixteen weeks for the first answer. That is why the shop’s quarter had no A/B test: the decisions it needed were whether to build, and the fake door answered that in a week.

The other rungs have their own arithmetic. Jakob Nielsen argued in March 2000 that five users find most usability problems in a design, and that three rounds of five beat one round of fifteen, because each round tests the fixes of the last. Line 5.3 is one round. The two customers who missed the changed price were enough to add a story, and nobody needed a confidence interval to see why.

Five products, and the test that would have been cheaper

Two of the cases below tested value before building, and two built first. The fifth is about the ideas a team ranks low. Each is mapped to the rung of the ladder that answered, or would have answered, its question.

Zappos and Dropbox — value tested before anything was built

In 1999 Nick Swinmurn wanted to know whether people would buy shoes without trying them on. Instead of building warehouses, he photographed shoes in local stores, posted the pictures online, and when someone ordered a pair, went back and bought it at full price. Every sale lost money and answered the value question with real orders. That is the concierge rung: the outcome delivered by hand, to find out whether anyone wants it before automating it.

In March 2008 Drew Houston posted a short demo video of Dropbox, aimed at the Digg community, before the product was ready for them. In his words, quoted in Eric Ries’s “The Lean Startup”, “our beta waiting list went from 5,000 people to 75,000 people literally overnight.” Joining a waiting list costs little, but it costs something, and seventy thousand people doing it is a fake door with a very wide frame.

Juicero and Quibi — built first, asked later

Juicero raised about $120 million to build a connected press that squeezed juice from its own sealed packs. Feasibility was answered thoroughly; it was an impressive machine. Value was never tested at the top rung. In April 2017 Bloomberg reporters squeezed the packs by hand and got nearly the same juice, which made the $400 press look unnecessary. On 1 September 2017 the company suspended sales of the press and the packs and offered refunds for 90 days.

Quibi launched on 6 April 2020 with $1.75 billion raised and a library of premium shows in chapters of ten minutes or less, made for phones on the move. The shutdown was announced on 21 October. The founders’ open letter gave two possible reasons: “because the idea itself wasn’t strong enough to justify a standalone streaming service or because of our timing.” After $1.75 billion, the company could not tell its value assumption apart from bad luck. A test designed to separate the two would have cost a sliver of that.

Bing — the idea ranked low

Ron Kohavi and Stefan Thomke tell this one in Harvard Business Review (2017). A Bing employee proposed a change to how ad headlines were displayed. It would take a few days to build, but it was one of hundreds of ideas, the program managers deemed it a low priority, and it “languished for more than six months”. Then an engineer ran a simple A/B test. Revenue rose 12% — more than $100 million a year in the United States alone — and it became the best revenue idea Bing had had. Experts are poor judges of which ideas will work, including their own.

Where it degrades — and what discovery cannot do

The honest reason discovery is skipped is that it looks like not shipping. A quarter spent building four features feels productive; a quarter spent testing eight ideas and building two feels slow — right up until someone counts what the four did. Kohavi and Thomke give the numbers from controlled experiments: “At Google and Bing, only about 10% to 20% of experiments generate positive results. At Microsoft as a whole, one-third prove effective, one-third have neutral results, and one-third have negative results.” Those are ideas good enough to reach an experiment.

• Interviews that ask what people want instead of what they last did.
• Discovery run to validate a decision already made — theatre with a method.
• A discovery phase with an end date, which turns it back into requirements gathering.
• Opportunities phrased as features, so every alternative is a variation of one idea.
• One solution per opportunity, so nothing was compared.
• Engineers brought in after the solution is chosen, which makes feasibility a late surprise.
• Tests with no criterion written in advance, which always pass.
• Recruiting by hand, which is why the rhythm dies in the first busy week.
• Research that would not change Monday, run anyway — procrastination with a method.

When to use it

Related pages

• ManagementRoadmappingRoadmapping is the work of deciding and communicating where a product is going, in what order and why, committing to outcomes and to very few dates. The hard part is not drawing the timeline but refusing to promise features on dates at the moment the team knows least about whether they will work. The page opens with one team’s finished roadmap, then covers what a roadmap is and the two ways a dated one goes wrong, Now/Next/Later, three readers and three versions, four ways to order the items, dates made after discovery, two public promises and two teams that changed the format, and where it degrades.
• ManagementMetrics & KPIA metric is a number with a written definition that a team steers by, and a KPI is one of the few metrics a team or a company has agreed to be judged on. The craft is building a chain from something the team can move this week to something the business needs this year, keeping every number honest under pressure, and reading it without fooling yourself. The page opens with one team’s finished set of metrics, then covers the words and the chain, writing a definition, Goodhart’s law and guardrails, one number at the top and the tree beneath it, reading a number, four numbers that misled their owners, and where it degrades.
• ManagementEpics, Features, TasksEpics, features and tasks are the levels of a work hierarchy, from a bet a leader makes for a quarter down to a step an engineer takes today. No standard defines them: the Scrum Guide uses none of the three words, and Jira, Azure DevOps and SAFe each put the epic at a different size. The page opens with every level on one screen and one epic taken down to tasks, then covers where the words come from, what each level is for, writing an epic as a bet with an exit, features, enablers and tasks, why rolled-up estimates mislead, five episodes from the hierarchy’s history, and where it degrades.