Testing LLM products

Twenty failures an LLM feature ships with — the check that catches each, and where it runs

Nobody opens a page about testing LLM products for a definition. They open it because a feature is due and the usual test plan does not fit: the output changes between runs, so what exactly does a test assert? The answer is less exotic than the question. Below are twenty failures an order assistant in an online shop is likely to ship with. It answers “where is my order”, explains the returns policy from the help centre and can start a return. Each row names what the customer gets, the check that catches it, what kind of check it is and when it runs. Rows 1–4 are the integration, 5–10 the output contract, 11–15 the content, 16–19 the adversary, and 20 is drift. Later sections cite them by number.

What actually changes: the missing oracle

Testing research has a name for the thing that broke. A test oracle is whatever decides whether an output is correct — usually an expected value written next to the input. In 2015 Earl Barr, Mark Harman and colleagues surveyed the oracle problem: how to tell correct behaviour from incorrect when that expected value is missing, or too costly to write. Scientific simulations have it, machine-learning models have it, and an LLM feature has it in its purest form. The expected output of “where is my order?” is any one of a thousand acceptable sentences.

What did not break is the rest of the idea. An oracle does not have to be an expected string. It can be a contract the output must satisfy, a relation between two outputs, a grader that returns a verdict, a person, or what real users did with the answer. Every row of the table above uses one of those, and the last column says which.

A common shortcut is to set the temperature to zero and assert the string anyway. Temperature zero means the model always takes its most likely next token, which should make the output repeatable. In September 2025 Horace He and colleagues at Thinking Machines Lab showed why it is not on an ordinary inference server. They sent one prompt a thousand times at temperature zero and got eighty different completions. All thousand agreed for the first 102 tokens; at the 103rd, 992 continued “Queens, New York” and eight “New York City”.

The cause was not randomness in the model but load on the server. How many other users’ requests were batched together changed the order of floating-point additions, and with it the last digits that settle a near tie between two tokens. Their fix, batch-invariant kernels, made all thousand completions identical, at a cost in speed. A shared APIAPIApplication Programming Interface — contract between programsThe rules one program follows to call another: which requests it may send, in what shape, and what comes back. To the caller it is a contract — everything behind it stays a black box, which is exactly what lets the other side be rewritten without breaking you.Full article →All terms does not run it for you. So treat a hosted model as non-deterministic whatever the temperature, and write tests that do not care.

The deterministic layer first: contract, tools and unhappy paths

Draw the feature as boxes and the shape of the work is obvious. The order assistant is a request handler, an intent router, a search over help-centre articles, a model call, a parser, three tools and a renderer. One box is probabilistic. The rest are ordinary code, and each carries exact checks that run in milliseconds. The diagram pins every row of the table to the box it tests.

Structured outputs removed one whole class of these failures and left three. When OpenAI introduced them in August 2024, its own evaluation of complex schemas went from under 40% for an older model to 100% with strict mode on. The same announcement lists what the guarantee does not cover. A refusal comes back as a refusal instead of the schema. A reply that reaches max_tokens stops before the object is complete. And the values inside a valid object can still be wrong. Those three limits are rows 1, 3 and 8, which is why a schema guarantee does not retire the tests — it tells you which ones to write.

The integration rows can run on every commit only if they cost nothing, and they cost nothing only if the provider is not called. So record real responses once — an ordinary answer, a refusal, a truncated reply, a tool calltool callthe model asking code to runA model response that, instead of text, names one of the functions you described to it and supplies arguments; your code runs it and hands the result back. The arguments are untrusted input and are validated like any request body — a model that has read an injected instruction produces a well-formed, entirely wrong call.Full article →All terms — and replay them from files in the test run. Then write the responses nobody can record on demand: a timeout, a 429, a 500, a tool that fails, a reply with broken JSONJSONJavaScript Object Notation — text format for dataA text format for passing data around: key–value pairs, lists, numbers and strings. It came out of JavaScript, but every language reads it now, and that is why most APIs answer in it rather than in anything more capable.All terms. The suite is now fast and deterministic, and it tests exactly the code that turns provider behaviour into what a customer sees.

No expected answer? Test a property

Rows 13–15 have no expected output, and they are still not judged by a score. They compare two runs of the feature with each other. Software testing has done this for decades wherever an oracle is missing. Metamorphic testing, reviewed by Tsong Yueh Chen and colleagues in 2018, checks a relation between outputs instead of the outputs themselves: rotate the input image, and the detected objects should rotate with it.

In 2020 Marco Tulio Ribeiro and colleagues brought the idea to language models as CheckList and named three test types. A minimum functionality test is a small set of simple cases that must all pass. An invariance test changes the input in a way that must not change the output. A directional test changes it in a way that must move the output in a known direction. In their user study, practitioners with CheckList wrote twice as many tests and found almost three times as many bugs as those without it.

A property needs no labelled answer, which is what makes it cheap to scale: one source case becomes ten follow-up cases by rule. Its limit is that both runs can be wrong in the same way. A question and its paraphrase can agree perfectly on a return window the policy never stated. That is why properties sit beside the scored rows rather than replacing them.

The adversary: the 2026 OWASP list as a test plan

Rows 15–19 come from a list a tester can use directly. OWASPOWASPOpen Worldwide Application Security Project — open application security foundationA non-profit foundation that publishes free, community-written security guidance — best known for its Top 10 lists of the most critical risks for web applications, APIs and LLM applications. A Top 10 ranks categories of risk to design and test against; it is not a checklist that proves a system safe.All terms published the 2026 edition of its Top 10 for LLM Applications on 4 August 2026. For the first time the ranking is not opinion alone: the practitioners’ vote carries three quarters of the weight, and 6,639 classified real incidents carry the rest. The preface states the posture behind all ten entries: “Stop trying to build a model that cannot be fooled. Build the system around it, so that when the model is fooled, and it will be, nothing important breaks.”

For a tester that sentence is the plan. You do not test whether the model can be tricked; it can. You test what the system lets a tricked model do — which tools it can reach, what it can show, what it can send and how much it can spend.

The other three entries — supply chain (LLM04), data and model poisoning (LLM05) and vector and embedding weaknesses (LLM09) — live mostly in how the model and the index are built and fed. They belong to a security review of that pipeline more than to the feature’s test suite. One practical trap: the 2026 list moved eight of its ten entries. System Prompt Leakage, LLM07 in 2025, is now Hidden Context Exposure, LLM08; Improper Output Handling fell from fifth to tenth. A test plan that cites entries by number alone now points at the wrong ones, so cite the year with the number — LLM08:2026.

Who writes what, and where each check runs

A tester on an ordinary feature writes steps and expected results. On an LLM feature the steps are trivial — send this, read that — and the difficulty moves into defining what a good answer is. That definition is a specification. It is genuinely hard to write, and it is the most valuable artefact on the project, because every scored row and every grader is only as good as it.

• QA owns the definition of good

  The written rubric: what must be present, what must never be, what tone, what to do when the answer is not known. Without it every review is an argument about taste.

• QA owns the failure corpus

  Every complaint, incident and surprising answer becomes a case. Curating it is what makes the suite better exactly as the product gets used.

• Engineering owns the harness

  Recording and replaying responses, pinning versions, storing results, wiring the gate into the pipeline. Ordinary infrastructure, and it should be treated as such.

• One named owner for the number

  Somebody must be able to block a release on quality. An LLM feature has no natural owner: the prompt, the search and the invoice usually belong to three different people.

A domain expert belongs in this loop and is usually missing from it. Whether a refund rule is stated correctly, whether a medical instruction is safe, whether a reply follows the company’s policy — the team that built the feature cannot settle these, and a rubric written without the expert encodes the engineers’ guess about the domain. Book the expert’s time early and explicitly, and treat their labels as the reference every grader is calibrated against.

Five releases, and the rows that would have caught each

None of these was a model that could not be tested. In each, something reached the public that one of the twenty rows checks for, and in every one the people who found it were users, journalists or outside researchers rather than the team’s own suite.

Where it degrades

The honest limit of everything on this page is that a suite checks what somebody thought to put in it. Twenty rows are twenty failures someone imagined; property tests need someone to invent the relation; a red team finds the attacks of its own members. Testing cannot show that an injection is impossible, which is exactly why the OWASP preface asks for a system in which a fooled model breaks nothing important — and why the rows that limit what the model can do (7, 16–19) are worth more than any score.

• Declaring the feature untestable and shipping on demos, which turns every customer into an unpaid tester with no way to report.
• Asserting exact strings, which gives a suite that fails on every run for reasons unrelated to quality — and gets switched off.
• Setting the temperature to zero and calling the output deterministic.
• Starting with the scored rows because they are new, and never reaching the exact ones that cause the incidents.
• A replay suite that has not been re-recorded since launch, green against a provider that no longer exists.
• A red-team exercise run once, its findings fixed and never turned into cases.
• Prompt edits and vendor upgrades that skip the suite because “nothing was deployed”.
• OWASP entries cited by number without the year, after the 2026 renumbering.
• Leaving cost and latency out of the suite, so the first person to notice is whoever opens the invoice.

When to use it

Related pages

• QAAI evaluationAI evaluation measures whether a model’s outputs are good enough to ship, with a fixed set of cases, a grader for each and numbers read with their uncertainty. The page opens with sixteen things a team will want to grade — the cheapest grader that settles each, and what fools it — then covers error analysis, building the set, the ladder of graders, model judges and how to trust one, reading the numbers, and four published measurements that misled.
• QAPrompt regressionPrompt regression testing checks, case by case, whether a change to a prompt or a model broke anything that worked before the change. The page opens with one prompt change reviewed from start to finish — the order assistant’s v7 to v8 — then covers negative flips, what to pin, three runs a case and the noise floor, the gate, the model upgrade, and four changes that went wrong.
• DevelopmentLLMOpsLLMOps is the practice of running a feature built on a language model in production, where the same input does not reliably produce the same output. It exists because everything that makes software releasable — a test that fails, a diff that explains a change, a rollback that restores yesterday — assumes that it does. The page opens with the twenty-four-line production checklist, then covers what each practice is replaced by, why the deployed unit is four things rather than one, evaluation as the test suite, what a trace has to contain, how cost and latency behave under real traffic, and five documented incidents.