Prompt regression

What a regression means when the answer is scored

In ordinary software a regression is unambiguous: a test that passed now fails. Here it needs defining, because a case can score 0.83 today and 0.79 tomorrow with nothing changed at all. The definition that works is per case and against a threshold: a case regressed when it passed its own criterion at the baseline and does not pass it now, measured across enough runs that sampling cannot explain the difference.

The diagram is the argument for this whole page. Both runs score thirteen out of sixteen, so a team reading the headline number would conclude the change did nothing and ship it. Case by case, two behaviours that used to work no longer do, and two others started working. Whether that is a good trade is a real question with a real answer — but it is a question, and the aggregate never asked it.

A comparison is only honest if everything else is pinned

The most common way a prompt experiment produces a wrong conclusion is that it was never a prompt experiment. Several things feed the output, they change on different schedules and by different hands, and unless each is fixed for the duration of the comparison you are attributing the sum of several changes to the one you happened to make.

The practical form of all this is to treat the prompt, the model pin, the retrieval index version and the tool schemas as one versioned artefact, and to record which version of that artefact produced every baseline. A prompt version on its own is not enough information to reproduce a result, and reproducing yesterday’s result is exactly what a regression suite has to be able to do.

Telling a real change from sampling noise

Even with everything pinned, the same case run twice can score differently, because generation samples from a distribution. Before comparing anything, measure how much your own numbers move when nothing changes: run the unchanged baseline three times and look at the spread. That spread is your noise floor, and any difference smaller than it is not a result, however much anyone wants it to be.

• Run each case several times — three is a workable minimum, five is better on a small set — and treat a case as passing only if it passes consistently rather than once.
• Set temperature to zero for evaluation runs where the product allows it. It does not make output deterministic, but it materially narrows the spread and makes small movements readable.
• Do the arithmetic on set size before believing a delta. On forty cases, one case is two and a half points, so a three-point move is one case changing its mind.
• Report the per-case diff as the primary artefact and the aggregate as a footnote, because the diff answers the question the aggregate cannot.

The gate: what runs, what blocks, what it costs

A regression suite that is not wired to a decision is a report. The arrangement that works treats a change to the prompt-model-index artefact exactly like a code change: it opens a pull request, the suite runs against it, and the result is a diff a reviewer reads before merging.

• What blocks

  Any protected case that regressed, any deterministic check failing, cost or latency outside budget. These are absolute and need no discussion.

• What warns

  An aggregate drop larger than the noise floor, or several regressions outside the protected set. A person decides, with the diff in front of them.

• What is recorded

  The full per-case result becomes the new baseline on merge, tagged with the artefact version. Tomorrow’s comparison needs today’s record.

• What it costs

  Ninety cases at three runs is a few minutes and a couple of dollars per pull request. Budget it explicitly, or somebody will quietly turn it off.

One drill deserves to be rehearsed rather than improvised: the model upgrade. A provider announces a new version and retires the old one on a date, and that is the single largest regression event an LLM feature will experience. Run the full suite against the new version while the old pin still works, read the per-case diff, fix what broke, and switch on a day you chose. Teams that discover this on the retirement date do the same work in an afternoon, with users watching.

Anti-patterns worth naming

• Editing the production prompt in an admin panel, which is a deployment with no diff, no review and no way back.
• Judging a change on the aggregate score, which is exactly the statistic that hides the cases that broke.
• Re-running only the results you dislike, which turns evaluation into selection and tunes the prompt to lucky seeds.
• Pinning to a moving alias, so the deployed system changes on the provider’s schedule rather than on yours.
• Keeping no baseline artefact, so "it used to work" is a memory rather than a file anybody can open.

When to use it

Related pages

• QAAI evaluationAI evaluation is how you find out whether a model's output is good enough to ship. An evaluation set is a fixed collection of inputs with a way to judge each output, run on every change and reported as rates. Building one is not hard. Building one that tells you the truth is, because two decisions decide everything: which cases are in it, and how each answer is graded. This page is about those two decisions — the ladder of graders from a free exact match to an expensive human, when a model can judge another model, and how to read a number without fooling yourself.
• QATesting LLM productsTesting an LLM product means testing a feature that can answer the same question differently twice, and every testing practice you have assumes the same input gives the same output. An LLM feature breaks that, and the usual reaction is to conclude that it cannot be tested — which is wrong, and expensively so. Most of what matters about an LLM feature is still exactly assertable, the part that is not becomes a measurement against a threshold, and the job of a tester shifts from writing steps to defining what a good answer is. This page is about that shift and what to do on the Monday after it.
• QARegression testingRegression testing is the practice of quickly catching a regression — something that used to work stopping working, when nobody meant to touch it. The suite it produces is best understood as institutional memory: a record of everything the system has already been taught, kept in a form that can be replayed. The hard parts are not writing the cases. They are deciding what earns a permanent place, what runs when the whole suite is too slow to run, and what to do about a case that fails for no reason.