Metrics & KPI

One team’s metrics on one screen — finished

Most people who look up metrics and KPIs want to see a set that works. Here is one: every number the customer team of the online shop watches, from the key result it is judged on down to what it deliberately ignores. It is the same shop as on the Decomposition, Release Process, Product Discovery and Roadmapping pages. The definition behind line 1.1 is written out in full two sections down. Later sections cite the lines by number.

1. 1.The outcome the team is judged on — one number

   1. 1.1Week-2 return visits: 18% before Reorder, 25% the target — key result 1.1 on the OKROKRObjectives and Key Results — a goal with measured resultsA goal-setting method that pairs one qualitative objective with two to five measurable key results proving it was reached, set and reviewed every quarter. It stops working when the key results become a task list or the scores feed into pay.Full article →All terms page; read every Monday, by weekly cohort; decides whether the reorder epicepica large backlog itemA large piece of work that is split into features or stories before anyone builds it, best written as a bet with a hypothesis and a condition that closes it. Its size depends on whose vocabulary is in use — anything above a story in Jira, a portfolio investment with a business case in SAFe — so check before comparing two epics.Full article →All terms continues

2. 2.Inputs the team can move within weeks — leading indicators

   1. 2.1Share of repeat orders started from Reorder or a saved basket — the epic’s leading indicator; below 20% four weeks after the full rollout, feature 3 is not built
   2. 2.2Median time from sign-in to a paid repeat order: 4 min 10 s now, under a minute the target — key result 1.2; always read with the 90th percentile beside it
   3. 2.3Share of order-history visits that tap Reorder: 14% on the fake door, watched as the rollout widens

3. 3.Guardrails — must not get worse while 1.1 moves; alerts, not targets

   1. 3.1Payment success at or above 97.5% in every week — committed key result 2.1; below it, no release stage opens
   2. 3.2Refunds under 2% of orders
   3. 3.3Support contacts at or below 6 per 100 orders
   4. 3.4Unsubscribes from reminder emails under 0.5% per send — from the day reminders exist

4. 4.Business KPIs — reported upward each month, moved by many teams at once

   1. 4.1Revenue from returning customers, per month
   2. 4.2Orders per returning customer per month: 1.2 → 1.4 — key result 1.3 as rewritten at week 6 on the OKR page
   3. 4.3Repeat orders delivered complete, per week — the company’s North Star; the tree under it is in the section on one number at the top

5. 5.Rules every number here obeys

   1. 5.1A written definition, an owner and one saved query — and the query is checked against the definition every quarter
   2. 5.2Each number names the decision it changes; a number that changes none is removed at the quarterly review
   3. 5.3No number on this list appears in anyone’s performance review
   4. 5.4Rates are read by cohort and durations by percentile — never a total or a bare average

6. 6.Deliberately not on it

   1. 6.1Registered customers in total — it can only go up, so it never signals a problem
   2. 6.2Page views and time on site — a slower checkout would raise both
   3. 6.3Story pointsstory pointsrelative size, not hoursA unit for estimating the relative size of backlog items — an 8 is roughly twice a 5 — deliberately not tied to hours. They are useful for one team’s planning and nothing else: turned into a target or compared across teams, they inflate instead of measuring.Full article →All terms delivered — the team’s planning unit, not a result

Metric, KPI, key result, North Star — and the chain from input to outcome

The words get used interchangeably, and most arguments about measurement are two people using the same word for different things. Each has a narrower meaning worth keeping apart, and each is one group of the hook.

Under the words sits a chain. An input is something a team directly controls this week. An output is what that work produced. An outcome is the change in behaviour or business result that followed. Robert Kaplan and David Norton made the same split for whole companies in 1992: financial measures “tell the results of actions already taken”, so a balanced scorecard adds operational measures that are “the drivers of future financial performance”.

The trap sits at both ends. Measure only inputs and you get a busy team with no evidence it matters. Measure only the outcome and you get a number that moves for reasons nobody can attribute — marketing spend, the season, a competitor’s outage — and a team that cannot tell whether its work helped. The hook has one outcome in group 1 and three inputs in group 2 for exactly that reason.

The same distinction is often described as leading and lagging indicators. A lagging indicator tells you what happened — revenue, churn, annual retention — accurately and too late to act on. A leading indicator moves earlier and predicts it imperfectly: line 2.1 moves in the first week of the rollout, line 1.1 takes a quarter. You steer by the leading ones and are judged on the lagging ones, and the whole difficulty is that the link between them is a hypothesis you keep re-checking. That is why 2.1 carries a threshold: below 20%, the hypothesis is treated as wrong and feature 3 is not built.

Write the definition before the dashboard

A metric’s name is not its definition. “Week-2 return visits” sounds precise and hides at least six decisions: who counts as returning, what counts as a visit, which week is the second, who is left out, where the data comes from, and when the number for a week is final. Line 5.1 makes each of them explicit. Here is line 1.1 written out in full, as the team keeps it next to the query.

Writing the definition is half of line 5.1. The other half is checking that the query computes what the definition says, because the two drift apart quietly: a join that drops guest checkouts, a filter added for one report and never removed, a timezone that moves Sunday night into Monday. Once a quarter the analyst counts one week’s cohort by hand and compares it with the query. It takes an afternoon. The Facebook case further down shows what two years without that afternoon looks like.

Goodhart’s law, and what a guardrail is for

Charles Goodhart, an economist at the Bank of England, wrote the original in 1975 about monetary targets: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” The version everyone quotes is the anthropologist Marilyn Strathern’s, from 1997: “When a measure becomes a target, it ceases to be a good measure.” Donald Campbell had reached the same point about social programmes, in the form later called Campbell’s law: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

This is not cynicism about people; it is arithmetic. Any metric is a proxy for something you cannot measure directly, and optimising hard enough always finds the gap between the proxy and the real thing. Steven Kerr’s 1975 title says it in six words: “On the Folly of Rewarding A, While Hoping for B”. The practical defence is the right-hand column below — a guardrail chosen as the thing that would break if the target were gamed.

It is worth being precise about who does the gaming, because the law is usually explained as if it were about dishonest people. Most metric distortion is done by conscientious people making locally reasonable choices — a support agent closing a ticket because the queue timer is red, an engineer splitting a change because the weekly deployment count is watched. The behaviour follows the measurement whether or not anyone intends it to. The VA case below is the extreme of the same mechanism, and it is why line 5.3 keeps every number on the list out of performance reviews.

One number at the top, and the tree beneath it

A single company-wide metric is genuinely useful for alignment and genuinely dangerous alone. Amplitude’s North Star Playbook asks that it represent “the value users get from your product” and be “a leading indicator of revenue”: nights booked rather than searches run, messages read rather than sent. For the shop it is line 4.3, repeat orders delivered complete each week — not orders placed, because an order that arrives half-empty is not value.

It fails when a team three layers down cannot influence it, at which point it stops being a goal and becomes weather. John Cutler put the test the other way round: “If you can move your North Star directly, it’s probably not a good North Star.” The fix is a tree beneath it. Break the number into the factors that multiply into it, and give each team the factor it can move.

The shop’s tree has four factors: returning customers who visit in the week, about 3,000; the share of those visits that start an order, 62%; the share of started orders that are paid, 96.8%; and the share of paid orders delivered complete, 95%. Multiplied, that is about 1,710 repeat orders a week. The customer team owns the first two, which is where 1.1 and group 2 sit. Payments owns the third — committed key result 2.1 is the same number seen from their side — and fulfilment owns the fourth. The company keeps one number; each team gets one it can change by Friday.

HEART and AARRR — checklists for finding candidates

Two frameworks help find the numbers in the first place. HEART comes from Kerry Rodden, Hilary Hutchinson and Xin Fu at Google (CHI 2010): Happiness, Engagement, Adoption, Retention, Task success, worked through a process they call Goals–Signals–Metrics — write the goal, name the user behaviour that would signal it, then pick the number. AARRR is Dave McClure’s “Startup Metrics for Pirates” (2007): Acquisition, Activation, Retention, Referral, Revenue, the stages of a customer’s life, so you can see which one leaks before optimising the wrong one. Applied to Reorder, HEART lands almost exactly on the hook.

Both are checklists for finding candidates, not sets to adopt whole. A team tracking all five HEART dimensions for every feature is producing a report, not steering — the Happiness row above is left empty on purpose, under line 5.2. Pick the one or two that would change a decision, and let the rest go.

Reading a number without fooling yourself

Most bad decisions made from metrics are not caused by the wrong metric. They are caused by the right metric read the wrong way, and four habits — line 5.4 and the rules for experiments — prevent almost all of it.

Cohorts, not snapshots

A snapshot mixes people who joined yesterday with people who joined three years ago, so a change in the mix looks like a change in behaviour. A cohort groups customers by when they arrived and follows that group over time, which separates “the product got better” from “we acquired different people”. That is why line 1.1 is read by weekly cohort. A retention number with no cohort behind it usually reflects last month’s marketing rather than this month’s product.

The retention curve is the most informative chart most products can draw. What matters is not where it starts but whether it flattens. A curve that levels off at 30% means a third of the people who arrive genuinely keep using the product, and growth compounds on that floor. A curve that keeps sliding towards zero means every new customer eventually leaves, and no amount of acquisition fixes it — you are filling a bucket with a hole in the bottom, faster. The two curves in the chart are almost identical for the first two weeks, which is exactly why a first-week number cannot tell them apart.

Percentiles, not the average

Averages hide everything interesting. An average checkout time of 70 seconds can mean that everyone waits 70 seconds, or that nine customers in ten take 40 seconds and one takes over five minutes because the changed-price screen confused them — and only the second is a problem, and only the second explains the support contacts. That is why line 2.2 is a median with its 90th percentile beside it. For anything a customer waits for, the tail is the experience people talk about.

Segments, before any total

A total can point the opposite way from every part of it. In autumn 1973 the University of California, Berkeley, admitted 44% of the 8,442 men who applied to graduate school and 35% of the 4,321 women. Peter Bickel, Eugene Hammel and J. W. O’Connell examined the departments one by one (Science, 1975) and called the aggregate “a clear but misleading pattern of bias against female applicants”. Women had applied in larger numbers to the departments that admitted fewest applicants of either sex. Pooled properly, the data showed “a small but statistically significant bias in favor of women”.

The shop meets the same effect every time its mix changes. When marketing’s first-order discount doubled new customers’ orders, the share of orders from returning customers fell while every returning customer was buying more — which is why key result 1.3 had to be rewritten as 4.2. Segment first, then average within the segment, never the reverse.

What an A/B result does and does not say

Two rules prevent most of the damage. First, decide the sample size and the duration before starting, and do not look at the result to decide when to stop. Evan Miller showed the cost in 2010: a test checked after every observation, up to 150, and stopped at the first “significant” result, finds an effect that is not there 26.1% of the time instead of 5%. Ramesh Johari of Stanford and colleagues from Optimizely published the formal treatment in 2017, with p-values that stay valid however often you look. Second, run at least one full week, so that weekday and weekend behaviour are both in the sample.

A “not significant” result is not a result of “no difference”. It means this test could not tell the effect from noise — the expected outcome when the effect is small, the sample modest, or both. The honest reading is “we still do not know”, and the decision then rests on judgement, cost and reversibility, as it did before the test. Ron Kohavi, Diane Tang and Ya Xu add a habit worth keeping from “Trustworthy Online Controlled Experiments” (2020): Twyman’s law, that any figure that looks interesting or different is usually wrong. Before celebrating, check that the groups are the sizes the split promised; a sample ratio mismatch means the experiment itself is broken.

Four numbers that misled their owners

Each case below is a number that said something different from what it seemed to say, and each is mapped to the line of the hook that would have caught it.

Facebook — a definition the query did not follow

Facebook defined the advertisers’ metric “average duration of video viewed” as total time spent watching a video divided by the total number of people who played it. The calculation actually divided by only the people who had watched for three seconds or more. Everyone who scrolled past in under three seconds vanished from the denominator, so the average came out longer. In September 2016 Facebook acknowledged the error on its help pages and replaced the metric; reports put the overstatement at 60–80%, for about two years. Facebook said billing was not affected. The definition was written down. What was missing was line 5.1’s second half — someone counting a sample by hand and comparing.

YouTube — a proxy replaced, twice

YouTube’s own account, by its VP of engineering Cristos Goodrow in September 2021, is Goodhart’s law told by the company it happened to. Recommendations first leaned on clicks, on the reasoning that “you wouldn’t click on something you don’t want to watch”. Then, in Goodrow’s words, “we learned back in 2011 that clicking on a video doesn’t mean you actually watched it” — a thumbnail promising match highlights could lead to someone in a bedroom talking about the match. In March 2012 Suggested Videos moved to watch time, and on 12 October 2012 search followed, “to reward engaging videos that keep viewers watching”. Goodrow: “When we first incorporated watchtime into recommendations, we saw an immediate 20% drop in views.” Watch time could be gamed too — not all of it is time people are glad they spent — so YouTube added “valued watchtime”, measured by surveys asking viewers to rate a video from one to five stars.

Two things make this the model case rather than a failure. The company accepted a fall in its headline number to fix the proxy, and it kept redefining the metric as the gap reappeared. That is line 5.2 at the scale of a whole company: the number is reviewed for whether it still means what it was chosen to mean.

Phoenix VA — a goal in the performance contract

The US Department of Veterans Affairs had a goal of seeing patients within 14 days, and it sat in staff performance contracts. The Inspector General’s report on the Phoenix VA Health Care System, published on 26 August 2014, found about 1,400 veterans on the official electronic wait list and over 3,500 more waiting to be scheduled, many of them on unofficial lists the measured number never saw. Of the 79 scheduling staff interviewed, thirty said they used the wrong desired date, “resulting in appointments showing a false 0-day wait time”, and eleven said they “fixed” or were told to “fix” appointments with waits over 14 days. The director’s claimed improvements in access were, in the report’s words, “inaccurate or unsupported”.

Nothing about the 14-day goal was wrong as a goal. It went wrong as a number people were paid against, measured by the people being paid. After the interim report in May 2014, the Acting Secretary removed the 14-day scheduling goal from employee performance contracts. Line 5.3 is that decision, taken before anyone is harmed rather than after.

Berkeley — the total and its parts

The Berkeley numbers in the section on reading are the fourth case, and the one with no villain at all. Nobody gamed anything and nobody miscounted. The aggregate was arithmetically correct and pointed the wrong way, because the mix of applications across departments differed by sex. It is the cleanest demonstration that a correct number read as a total can still be the wrong evidence, and it is why line 5.4 asks for segments before any total.

Where it degrades — and what a number cannot tell you

A metric earns its place by changing a decision. The test is direct: for each number on a dashboard, name the decision that would go differently if it moved by 20%. Most dashboards lose two-thirds of their contents to that question, and the team is better off for it — a wall of numbers nobody acts on trains everyone to ignore the ones that matter.

• Vanity metrics — totals that only go up, so they can never signal a problem.
• A target with no guardrail, which is an open invitation to game it.
• A guardrail turned into a second target, and gamed in turn.
• A definition nobody wrote down, so two teams argue about a trend in two different numbers.
• A definition written down and a query that never gets checked against it.
• Snapshots instead of cohorts, so a change in who arrived is read as a change in the product.
• Averages hiding the distribution — a median and a 90th percentile tell different stories.
• Stopping an A/B testA/B testa controlled experiment on live usersAn experiment that shows two versions of something to randomly split groups of users at the same time and compares a chosen metric between them. It is the strongest evidence of an effect and the dearest: telling a rate of 18% from 19% takes about 24,000 users in each group.Full article →All terms the moment it looks good, which manufactures wins out of noise.
• Metrics tied to individual pay or appraisal, which corrupts the number within one review cycle.

When to use it

Related pages

• ManagementOKROKR — Objectives and Key Results — is a goal-setting method that pairs one qualitative objective with two to five measurable results proving it was reached. It came from Intel in the 1970s and reached Google in 1999, and it works only when key results measure outcomes rather than tasks, when committed and aspirational goals are scored by different rules, and when scores stay out of pay. The page opens with one team’s OKRs for a quarter, finished, then covers what Grove, Doerr and Google actually wrote, outcomes and guardrails, how a score is computed, alignment and the weekly cycle, OKR beside KPI and pay, four cases and where it degrades.
• ManagementProduct DiscoveryProduct discovery is the continuous work of deciding what is worth building, by testing ideas with real customers before engineering time is spent on them. It is not a phase before delivery and not requirements gathering: a team talks to customers every week and maps what it hears as opportunities, so solutions compete rather than arrive decided. The page opens with one finished quarter of discovery, then covers discovery beside delivery, the opportunity solution tree, interviews, the four risks and assumption mapping, tests from cheapest to dearest, five products and the test that would have been cheaper, and where the practice degrades.
• ManagementRoadmappingRoadmapping is the work of deciding and communicating where a product is going, in what order and why, committing to outcomes and to very few dates. The hard part is not drawing the timeline but refusing to promise features on dates at the moment the team knows least about whether they will work. The page opens with one team’s finished roadmap, then covers what a roadmap is and the two ways a dated one goes wrong, Now/Next/Later, three readers and three versions, four ways to order the items, dates made after discovery, two public promises and two teams that changed the format, and where it degrades.