AI evaluation

Sixteen things to grade — the cheapest grader that settles each, and what fools it

People come to a page about AI evaluation with a concrete question: how do I grade this? The answer is almost never “ask another model”. Below are sixteen things a team grades on the order assistant from Testing LLM products — the shop’s chat that answers “where is my order”, explains the returns policy and starts returns. For each: the cheapest grader that settles it, and what fools that grader. The prefix names the rung on the ladder further down. Later sections cite the rows by number.

Read before you measure: error analysis

The usual first move is to pick metrics — helpfulness, faithfulness, a hallucinationhallucinationconfident invented outputA model stating something invented with the same fluency it states something true. It is not a bug to be patched out: the model predicts plausible text, and plausible is not the same as correct — which is why anything checkable gets checked.Full article →All terms score — and wire them into a dashboard. It produces numbers about problems nobody has seen. Hamel Husain and Shreya Shankar, who have taught evaluation to thousands of engineers, put it bluntly: generic metrics are not useful in most AI applications. Start instead by reading what the system actually did.

The method comes from qualitative research and has three steps. Read at least a hundred real conversations, and for each one that went wrong, write a short note in your own words — open coding. Then group the notes into failure categories and count them — axial coding. Stop when new conversations stop producing new kinds of failure, which researchers call saturation. Only then decide how each category will be graded.

Two things in that table are typical. The biggest group is not the one anybody guessed before reading, and most groups turned out to be checkable by code once somebody had looked at them closely. A team that had started from a “helpfulness” score would have measured neither the wrong order nor the reply in the wrong language.

What goes in the set, and how big it needs to be

The instinct is to build for coverage: a case per feature, spread evenly. That produces a set that scores ninety-something per cent from the first day and never moves again — so it can no longer tell you anything. A set earns its keep by being concentrated where the system is weak, and its composition should be deliberately unbalanced. Here is the order assistant’s.

Thirty to fifty real failures is a genuine starting point; a hundred and fifty is a mature set for one feature. The held-out part matters more than its size suggests. Without it you eventually tune the prompt to the set instead of to the customers, and nothing in the numbers will tell you that it happened. And every case needs a label from someone who knows the domain — for the order assistant, the person who owns the returns policy, not the engineer who wrote the prompt.

The ladder of graders

How an answer is judged matters more than how many answers you judge. There is a clear order to try graders in, and the discipline is to climb only when the rung below genuinely cannot settle the question.

The second rung covers far more of a real set than teams expect. If the answer quotes a number, recompute it (row 3). If it cites a passage, find the passage (row 4). If it calls a tool, assert the arguments (row 5). Each is deterministic code that runs in milliseconds, costs nothing and produces a result nobody can argue with — which is a different kind of thing from a score.

The middle rung is the one to distrust. Word-overlap metrics such as BLEU and ROUGE compare a reply with an approved one, which suits machine translation of fixed sentences and little else. In 2016 Chia-Wei Liu and colleagues found that on dialogue replies they agree with human judgement weakly or not at all (row 12). And running code against tests (row 13) is only as good as the tests: in February 2026 OpenAI found that most of the hard problems in a widely used coding benchmark had tests that reject correct solutions — the fourth case at the end of this page.

A model as a judge, and how to trust one

A model can grade another model’s output, and for questions about meaning it is genuinely useful at a scale people cannot reach. In 2023 Lianmin Zheng and colleagues found that strong judges agreed with human preferences more than 80% of the time — about as often as people agree with each other. The same paper named the judge’s biases, and later work added more. A judge is a measuring instrument with known faults, and each fault has a known counter.

Then calibrate, and calibrate the way you would test any classifier. Have the domain expert label cases with pass or fail and a one-line reason. Husain recommends about a hundred labelled examples per failure kind, warning that below sixty the uncertainty is too wide to conclude anything. Split them: a few go into the judge’s prompt as examples, about half are a development set you improve the prompt against, and the rest are a test set you look at once, at the end.

On the test set, report two numbers rather than one. The true positive rate (TPRTPRtrue positive rate — share of real failures caughtThe share of real positives a test, a classifier or a model judge actually catches — the same quantity as recall. A TPR of 80% means one real failure in five gets through; read it beside the TNR, because a judge that flags nothing has a perfect TNR and a TPR of zero.All terms) is the share of real failures the judge catches; the true negative rate (TNRTNRtrue negative rate — share of good cases passedThe share of real negatives — good answers, healthy cases — that a test, a classifier or a model judge correctly lets pass. A TNR of 91% means about one good case in eleven is flagged by mistake; a high TNR on its own says nothing, since passing everything scores 100%.All terms) is the share of good answers it passes. Agreement alone hides the failure that matters, because most answers are good: a judge that passes everything agrees with the expert as often as good answers occur.

Cohen’s kappa, a statistic from 1960, says the same thing in one number: agreement corrected for what chance would give. Judge A’s kappa is 0 and judge B’s is 0.62. And once TPR and TNR are known, the judge’s raw pass rate can be corrected. Judge B passes 80% of answers; knowing it fails 9% of good answers and lets 20% of bad ones through, the true pass rate works out at 85%.

The judge is also a dependency with a version and a price. Pin its model, or a provider update quietly re-scales every historical number. Keep its prompt in version control beside the code, because changing the ruler is as significant as changing what it measures, and far easier to do by accident. Re-check TPR and TNR whenever either the judge or the system under test changes much.

Reading the numbers without fooling yourself

An evaluation result is a measurement from a sample, and a measurement without its uncertainty is half a fact. In 2024 Evan Miller set out how to report evaluation results the way other sciences report experiments: a standard error beside every score, clustered errors when cases come in related groups, repeated sampling to cut noise, comparisons made question by question, and a check before the run that the set can detect the difference you care about.

Telling a 3-point change from noise on an 82% pass rate takes about 630 independent cases. Few teams have that, and they do not need it, because two of Miller’s recommendations do most of the work. Compare two versions case by case — the same case, before and after — rather than one average against another; that removes the variation between cases and costs nothing. And when cases come in groups, such as several turns of one conversation, use clustered errors, which in his examples ran to more than three times the naive ones.

Other people’s numbers: four measurements that misled

Public benchmarks and vendor figures are evaluations too, built by people under the same pressures, and they fail in the same ways. Each of these four was published, widely cited and wrong about what it seemed to measure. None of them involved fraud in the ordinary sense; each is a decision about how to build or grade a set.

Where it degrades

The honest limit is that every number on this page measures agreement with somebody’s idea of good, written down at one moment. The rubric drifts as people read, the expert labels from last quarter’s policy, and the set describes the failures you have already met. An evaluation can tell you that a change did not make known things worse. It cannot tell you that the feature is good, and the sixteenth row — what customers actually did — stays the only one that answers that.

• Generic metrics chosen before anyone read a conversation, measuring problems nobody has seen.
• A set built from cases that already pass, which reports a high number and detects nothing.
• A model judge on questions code could settle, buying uncertainty at a price per case.
• A judge reported by its agreement rate, with nobody checking whether it catches any failures.
• One holistic score out of ten, averaging away the one dimension that broke.
• No holdout, so the prompt is tuned to the set until the set stops predicting anything.
• A 3-point gain on forty cases from one run, announced as a result.
• A public leaderboard position taken as evidence about your own use case.
• An unpinned judge, which quietly re-scales every historical result.

When to use it

Related pages

• QATesting LLM productsTesting an LLM product means testing a feature whose model answers the same question differently on two runs, while most of its checks remain exact assertions. The page opens with twenty failures an order assistant ships with — the check that catches each, and where it runs — then covers the missing test oracle, the deterministic layer and its unhappy paths, properties for answers with no expected value, the 2026 OWASP list as a test plan, who owns what, and five releases that went wrong.
• QAPrompt regressionPrompt regression testing checks, case by case, whether a change to a prompt or a model broke anything that worked before the change. The page opens with one prompt change reviewed from start to finish — the order assistant’s v7 to v8 — then covers negative flips, what to pin, three runs a case and the noise floor, the gate, the model upgrade, and four changes that went wrong.
• DevelopmentLLMOpsLLMOps is the practice of running a feature built on a language model in production, where the same input does not reliably produce the same output. It exists because everything that makes software releasable — a test that fails, a diff that explains a change, a rollback that restores yesterday — assumes that it does. The page opens with the twenty-four-line production checklist, then covers what each practice is replaced by, why the deployed unit is four things rather than one, evaluation as the test suite, what a trace has to contain, how cost and latency behave under real traffic, and five documented incidents.