AI evaluation

What goes in the set, and how big it needs to be

The instinct is to build for coverage: a case per feature, a case per category, spread evenly. That produces a set that scores ninety-something per cent from the first day and never moves again, which means it can no longer tell you anything. An evaluation set earns its keep by being concentrated where the system is weak, so its composition should be deliberately unbalanced.

• Cases from production

  Every complaint, every negative rating, every surprising answer. This is the highest-value source and the only one that keeps growing as the product is used.

• Cases nobody can get right

  Questions whose answer is not in the source. The expected behaviour is to say so, and this is the category most systems quietly fail.

• Cases that must never break

  A protected set: refusals, regulated wording, the exact behaviour a customer was promised. Any regression here blocks a release regardless of the average.

• A few easy ones

  Not for the score — as a canary. When the easy cases start failing, something structural has broken and the interesting numbers are meaningless.

Thirty to fifty cases is a real starting point and a hundred is a mature set for one feature. The value is not in the count — it is in each case being one somebody actually got wrong. And it is worth splitting the set from the start: a working set you look at and iterate against, and a holdout you run less often and never tune towards. Without the holdout you eventually optimise the prompt for the evaluation rather than for the users, and you will not be able to tell that it happened.

The ladder of graders

How an answer is judged matters more than how many answers you judge. There is a clear order to try things in, and the discipline is to climb only when the rung below genuinely cannot settle the question.

The second rung deserves more attention than it usually gets, because it covers far more of a real evaluation set than teams expect. If the answer quotes a number, recompute it and compare. If it cites a passage, check that the passage exists in the source document and actually contains the claim. If it calls a tool, assert the arguments. If it is a summary, check that every named entity in it appears in the input. Each of those is deterministic code, runs in milliseconds, costs nothing, and produces a result nobody can argue with — which is a materially different thing from a score.

Using a model as a judge, properly

A model can grade another model’s output, and it is genuinely useful for subjective criteria at a scale humans cannot reach. It is also a measurement instrument that has to be validated before its readings mean anything, and most teams skip that step entirely.

• Ask for a binary verdict against a written criterion, not a score out of ten. "Does the answer cite a source for every factual claim — yes or no" is answerable and repeatable; "rate the quality from one to ten" produces sevens and eights forever.
• Split a rubric into several independent binary checks rather than one holistic judgement. You then know which dimension moved, which is the entire reason for measuring.
• Give the judge the reference material it needs to be right — the source document, the expected answer, the policy — rather than asking it to know the domain.
• Watch for position bias when comparing two answers: the same pair judged in the opposite order should give the opposite verdict, and if it does not, run both orders and average.
• Do not let the model that generated the answer grade the answer. It is measurably kinder to its own output, and you have built an instrument that reports what you hoped.

Then calibrate. Have a person label fifty cases against the same written criterion, run the judge on the same fifty, and measure how often they agree. Below rough agreement the judge is not measuring your criterion and the rubric needs rewriting; at high agreement you can use it at scale and re-check quarterly. Report that agreement number alongside every evaluation result, because a score of 0.86 from an uncalibrated judge is not a measurement — it is a rumour with a decimal point.

Reading the numbers without fooling yourself

An evaluation produces rates, not verdicts, and rates need care. Two questions decide whether a reported difference means anything: how many cases were behind it, and how much the number moves when nothing changes.

Anti-patterns worth naming

• A set built from cases that already pass, which reports a high number and detects nothing.
• A single holistic score out of ten from a judge, which averages away the one dimension that broke.
• No holdout, so the prompt is tuned towards the evaluation until the evaluation stops predicting anything.
• An unpinned judge model, which quietly re-scales every historical result the next time the provider ships an update.
• A set that is never grown from production, which describes last quarter’s users and nobody else.

When to use it

Related pages

• QATesting LLM productsTesting an LLM product means testing a feature that can answer the same question differently twice, and every testing practice you have assumes the same input gives the same output. An LLM feature breaks that, and the usual reaction is to conclude that it cannot be tested — which is wrong, and expensively so. Most of what matters about an LLM feature is still exactly assertable, the part that is not becomes a measurement against a threshold, and the job of a tester shifts from writing steps to defining what a good answer is. This page is about that shift and what to do on the Monday after it.
• QAPrompt regressionPrompt regression testing is how you find out whether editing a prompt has made the feature worse. A prompt is a piece of production configuration that anyone can edit in a text box, that has no type system, no compiler and no diff anybody can read, and that changes the behaviour of the whole feature. Regression testing it is therefore not optional — but it cannot be equality checks, because the output moves on its own. This page covers what a regression means when results are scored, what has to be pinned before any comparison is honest, how to tell a real change from sampling noise, and what should block a release.
• DevelopmentLLMOpsLLMOps is the practice of running a feature built on a language model in production, and it exists because everything that makes software releasable — a test that fails, a diff that explains a change, a rollback that restores yesterday — assumes the same input gives the same output. LLM features break that assumption, so the practices have to be rebuilt around it. This page covers what replaces the test suite, what the deployable artefact actually is, what to record on every call, and how cost and latency behave once real traffic arrives.