Testing LLM products

What actually changes, and what does not

One thing changes: you can no longer assert that the output equals an expected string. Everything else that follows — that a single run proves little, that results are read as rates, that you need a set of cases rather than a case — is a consequence of that one fact rather than a new discipline.

The right-hand panel is where the attention goes, and the bottom row is where the work should start. Before any score is involved, a large share of what a product owner actually cares about can be checked exactly: that the response is valid JSON matching the schema the interface expects, that the classification is one of the four allowed labels, that the tool call carried the right arguments, that no personal data appears in the text, that the answer arrived within the latency budget and under the cost ceiling. These are ordinary tests with ordinary assertions, and they belong to the same suite as everything else you test.

Build the deterministic layer first

Before writing a single scored evaluation, cover the things that have exact answers. They are cheap, they run in milliseconds, they never disagree with themselves, and each one closes a failure mode that would otherwise reach a user.

The role shift: who writes what

A tester on a conventional feature writes steps and expected results. On an LLM feature the steps are trivial — send this, read that — and all the difficulty moves into defining what a good answer is. That definition is a specification, it is genuinely hard to write, and it is the single most valuable artefact on the project.

• QA owns the definition of good

  The written rubric: what must be present, what must never be, what tone, what to do when the answer is not known. Without it, every review is an argument about taste.

• QA owns the failure corpus

  Every complaint, incident and surprising answer becomes a case. Curating this is the work that makes the suite get better exactly as the product gets used.

• Engineering owns the harness

  Running the set, pinning versions, recording results, wiring the gate into the pipeline. This is ordinary infrastructure and should be treated as such.

• One named owner for the number

  Somebody must be able to block a release on quality. An LLM feature has no natural owner — the prompt, the retrieval and the invoice usually belong to three different people.

A domain expert belongs in this loop and is usually missing from it. Whether a legal summary is correct, whether a medical instruction is safe, whether a support answer follows policy — none of these can be settled by the team that built the feature, and a rubric written without that expert encodes the engineers’ guess about the domain. Book the expert’s time explicitly, early, and treat their labels as the reference every automated grader is calibrated against.

Where each check belongs in the pipeline

Scored evaluations cost money and take minutes, so they cannot run on every commit the way unit tests do. That constraint produces a natural tiering, and writing it down stops the two failure modes: a suite so expensive it is quietly disabled, and a release process where nothing checks quality at all.

The failures worth testing for on purpose

Some failure modes belong to this class of product specifically, and none of them will be found by testing the happy path. Each deserves a small permanent set of cases.

• Instructions arriving inside content the model reads. A document, an email or a web page can contain text aimed at your system, and if retrieved content is treated with the same authority as your prompt, it can be. Test with hostile documents on purpose.
• The confident wrong answer. Ask questions whose answers are not in the source at all; the correct behaviour is to say so, and the default behaviour is often to produce something plausible instead.
• A model version that changed under you. Provider aliases move; if your pin is an alias, your system upgraded itself in production with no diff, no review and no rollback path.
• Long inputs. Behaviour at the edge of the context window differs from behaviour in the middle of it, and truncation usually removes the instruction rather than the noise.
• Languages and formats other than the ones in the demo. A product tested only in English will meet its first Ukrainian address, its first PDF table and its first emoji on the day it launches.

Anti-patterns worth naming

• Declaring the feature untestable and shipping on demos, which converts every customer into an unpaid tester with no reporting channel.
• Asserting on exact strings, producing a suite that fails on every run for reasons unrelated to quality.
• Judging a change from one run of one prompt, where sampling alone can produce either conclusion.
• Building an evaluation platform before there is a feature and a handful of real failures to put in it.
• Leaving cost and latency out of the test suite, so the first person to notice is whoever opens the invoice.

When to use it

Related pages

• QAAI evaluationAI evaluation is how you find out whether a model's output is good enough to ship. An evaluation set is a fixed collection of inputs with a way to judge each output, run on every change and reported as rates. Building one is not hard. Building one that tells you the truth is, because two decisions decide everything: which cases are in it, and how each answer is graded. This page is about those two decisions — the ladder of graders from a free exact match to an expensive human, when a model can judge another model, and how to read a number without fooling yourself.
• QAPrompt regressionPrompt regression testing is how you find out whether editing a prompt has made the feature worse. A prompt is a piece of production configuration that anyone can edit in a text box, that has no type system, no compiler and no diff anybody can read, and that changes the behaviour of the whole feature. Regression testing it is therefore not optional — but it cannot be equality checks, because the output moves on its own. This page covers what a regression means when results are scored, what has to be pinned before any comparison is honest, how to tell a real change from sampling noise, and what should block a release.
• DevelopmentLLMOpsLLMOps is the practice of running a feature built on a language model in production, and it exists because everything that makes software releasable — a test that fails, a diff that explains a change, a rollback that restores yesterday — assumes the same input gives the same output. LLM features break that assumption, so the practices have to be rebuilt around it. This page covers what replaces the test suite, what the deployable artefact actually is, what to record on every call, and how cost and latency behave once real traffic arrives.