Prompt regression

A prompt change on one screen — the order assistant’s v7 to v8, reviewed

Nobody opens a page about prompt regression for a definition. They open it because a prompt change is waiting in review and the question is whether it can ship. Below is one such change, reviewed from start to finish. The feature is the order assistant from Testing LLM products and AI evaluation: the shop’s chat that answers “where is my order”, explains the returns policy and starts returns. The change is one sentence. The groups follow the order of the work — the change, what was pinned, the run, the per-case diff, the verdict, what happened next — and later sections cite the lines by number.

1. 1.The change

   1. 1.1One sentence added to the system promptsystem promptthe standing instructionsThe part of a prompt carrying the role and the rules, sent on every call and normally not shown to the user. It is not a secret: it sits in the same window as everything else and has been extracted from production systems by asking, so anything genuinely confidential belongs outside it.Full article →All terms, v7 → v8: “Always tell the customer the date their return window closes.”
   2. 1.2The reason: the largest failure group in error analysis — 13 of 41 notes left out the condition that decides (AI evaluation, row 7; Testing LLM products, row 12).
   3. 1.3A pull requestpull requesta change proposed for reviewA request to merge one branch into another, where the change is reviewed, discussed and checked by the pipeline before it reaches the main branch; GitLab calls the same thing a merge request. Its cost is waiting: a pull request that sits unreviewed for days is where long-lived branches and painful merges come from.Full article →All terms against prompts/order-assistant.md, reviewed like code. The owner of the returns policy approves the wording.

2. 2.Everything else, pinned — identical on both sides

   1. 2.1The model: the same dated snapshot as v7’s baseline — never an alias.
   2. 2.2Sampling: the production settings, temperature 0.3, so the run measures what customers will get.
   3. 2.3The help-centre index: the build of Monday 5 October, frozen for the run.
   4. 2.4The tools: get_order, start_return and handoff at schema v3, their responses replayed from recordings, so both versions see the same orders.
   5. 2.5The judge: its model and prompt pinned; TPRTPRtrue positive rate — share of real failures caughtThe share of real positives a test, a classifier or a model judge actually catches — the same quantity as recall. A TPR of 80% means one real failure in five gets through; read it beside the TNR, because a judge that flags nothing has a perfect TNR and a TPR of zero.All terms 80% and TNRTNRtrue negative rate — share of good cases passedThe share of real negatives — good answers, healthy cases — that a test, a classifier or a model judge correctly lets pass. A TNR of 91% means about one good case in eleven is flagged by mistake; a high TNR on its own says nothing, since passing everything scores 100%.All terms 91% on held-out labels (AI evaluation).
   6. 2.6The set: the 120 working cases. The 30 held-out casesheld-out setexamples never trained onExamples separated out before training or tuning begins and never used to change the thing being measured. Without one there is no way to tell a model that learned the task from one that memorised the examples — and the second scores perfectly on everything anybody looked at.Full article →All terms wait for the release check.

3. 3.The run

   1. 3.1Every case three times on each version: 720 replies, plus the judge’s calls on the scored cases — 11 minutes, about $4.
   2. 3.2Each case gets a state per version: stable pass (3 of 3 runs), stable fail (0 of 3) or unstable (1 or 2).
   3. 3.3The noise floor, measured first: v7 against a second run of v7. No case moved between stable pass and stable fail; 5 moved in or out of unstable.

4. 4.The per-case diff, v7 → v8

   1. 4.1Fixed, stable fail → stable pass: 9 cases, 8 of them in the targeted group.
   2. 4.2Broke, stable pass → stable fail: 4 cases. Two are protected — final-sale items, now told “you can return it until 24 October”.
   3. 4.3The other two breaks are questions the policy does not cover. v8 answers them with a date instead of handing off (Testing LLM products, row 10).
   4. 4.4Unstable on either side: 14 cases — listed, not counted. Unchanged: 93.
   5. 4.5The headline number: stable passes 92 → 99 of 120, 76.7% → 82.5%.

5. 5.The verdict

   1. 5.1Blocked. Two protected cases broke, and any protected case that breaks blocks the release, whatever the average did.
   2. 5.2It would have warned anyway: two more breaks outside the protected set, against a noise floor of zero.
   3. 5.3Read by a person before deciding: the 13 conversations that flipped, about 25 minutes. All four breaks were real. One of the nine fixes was not — the judge passed a date taken from the customer’s other order.
   4. 5.4On the dashboard alone, v8 was 5.8 points better and would have shipped.

6. 6.What happened next

   1. 6.1v9 rewrites the sentence: “When the item can be returned, tell the customer the date its return window closes.” The handoff rule stays above it.
   2. 6.2v9 against v7: 9 fixed, 0 broke, all 20 protected cases pass, 11 unstable.
   3. 6.3The 30 held-out cases, run once before release: nothing broke.
   4. 6.4Merged on Thursday 15 October 2026. v9’s per-case results are stored as the new baseline, tagged with every version in group 2.
   5. 6.5Two new protected cases: a final-sale item asked about in the words v8 failed on, and the same question in Ukrainian. The protected set is now 22.

What counts as a regression when the output moves: a negative flip

In ordinary software a regression is unambiguous: a test that passed now fails. With a model in the loop the same case can pass on one run and fail on the next with nothing changed, so the word needs a definition. Machine-learning research already has one. In 2020 Sijie Yan and colleagues called it a negative flip: a sample the old model got right and the new model gets wrong. They also made the point this page rests on — that reducing such inconsistencies between versions can matter as much in practice as reducing the overall error.

In 2024 Jessica Echterhoff and colleagues at Apple measured the same thing on language models. When a base model was updated, task adapters trained on top of it in exactly the same way started getting previously correct answers wrong — although nothing in their own training had changed. So a prompt regression suite does one job. It lists every case that flipped, in each direction, and treats a flip from pass to fail as a finding regardless of what the total did.

Read the diagram as a table of moves. Rows are the state of a case on v7, columns its state on v8. The diagonal holds cases that stayed where they were; below it are the fixes, above it the breaks. The grey row and column are cases that were unstable on one side, which the section after next explains. The headline number sees only the column totals, so it cannot tell nine fixes and four breaks from five fixes and none.

Prompts make flips cheap to cause. In 2023 Melanie Sclar and colleagues changed only the formatting of prompts with worked examples — separators, spacing, capitalisation, nothing a reader would call a change of meaning — and measured accuracy differences of up to 76 points on one model. They also found that the best format for one model was a weak guide to the best format for another. A one-sentence edit is a bigger change than any of theirs.

Pin everything you are not testing

A prompt experiment most often reaches a wrong conclusion because it was never a prompt experiment. Six things feed the order assistant’s replies. They change on different schedules and in different hands, and only one of them was in the pull request. Unless the other five are held still for the run, the diff credits the sum of several changes to the one somebody happened to make.

Two rows deserve a sentence each. Sampling is pinned to the production values rather than to zero: temperature zero does not make a hosted model deterministic anyway (Testing LLM products explains why), and a run at settings customers never see measures a different system. And the tools are replayed rather than called. An order that moved from “packed” to “shipped” between the v7 run and the v8 run is a real difference in the input, and in the output it looks exactly like a regression.

The prompt itself belongs in your repository, next to the code that sends it. Prompt managers in vendor consoles make editing easy and history hard, and they can disappear. On 3 June 2026 OpenAI deprecated its stored prompt objects and its Evals platform together, set both to shut down on 30 November 2026, and advised moving prompt content into application code. A prompt and a suite that live in your repository survive a vendor’s product decisions.

Three runs a case: telling a flip from noise

Even with all six pinned, the same case can pass on one run and fail on the next, because generation samples from a distribution. One run per case therefore cannot tell a flip from a coin that landed the other way. The fix is cheap: run every case three times and give it a state. Three passes make a stable pass, three failures a stable fail, and anything in between is unstable. Only a move between the two stable states counts as a flip.

Before comparing two versions, compare one version with itself. Run the unchanged baseline a second time and diff it against the first, exactly as you would diff a candidate. This is an A/A comparison, and whatever it shows is your noise floor. For the order assistant it showed no flips and five cases drifting in or out of unstable (line 3.3). A candidate that breaks four stable cases is therefore four findings, not four coincidences.

Unstable cases are information, not noise to throw away. A case that passes two runs in three is one where the model sits close to a decision boundary, and that is exactly where the next small edit will push it over. Keep the unstable list next to the lists of flips. If a case stays unstable three releases running, read it — the cause is usually an ambiguous rule in the prompt or in the policy itself. The rest of the statistics — the error bar on a pass rate, clustered errors, why comparing case by case beats comparing averages — is on the AI evaluation page.

The gate: what blocks, what warns, what a person reads

A regression suite that is not wired to a decision is a report. The arrangement that works treats a change to any of the six inputs exactly like a code change. The change opens a pull request, the suite runs against it, and the result is a diff a reviewer reads before merging. Which results stop the merge on their own is agreed before the run, not argued about after it.

The protected set is the part of the suite that blocks on its own, so how a case gets in matters. Four reasons qualify: a failure that already reached customers, a statement the company is bound by, wording a regulator cares about, and anything that touches safety. The order assistant’s final-sale cases are protected for the second reason. When a chatbot misstated Air Canada’s bereavement-fare policy, a British Columbia tribunal ruled in February 2024 that the airline was responsible for what its chatbot said. An assistant that tells a customer a final-sale item can be returned has made the shop a promise.

A case leaves the protected set only by a decision someone signs, never by being quietly deleted when it becomes inconvenient. The set grows from incidents, so expect it to grow: the order assistant’s went from 20 to 22 in one review (line 6.5). When a protected set gets so large that every change breaks something in it, the problem is usually the prompt’s structure rather than the set — too many rules competing in one paragraph.

The model upgrade: the biggest regression run you will do, so rehearse it

Every pinned snapshot has a retirement date, and the provider sets it. OpenAI’s published minimum is six months’ notice for a generally available model and three months for specialised variants; preview models can go much faster, and its documentation gives two weeks as an example. Anthropic promises at least 60 days for publicly released models. On 14 October 2026 OpenAI told developers that its GPT-4o and GPT-4.1 snapshots leave the APIAPIApplication Programming Interface — contract between programsThe rules one program follows to call another: which requests it may send, in what shape, and what comes back. To the caller it is a contract — everything behind it stays a black box, which is exactly what lets the other side be rewritten without breaking you.Full article →All terms on 14 January 2027. For a feature pinned to one of them, this section has a date attached.

An alias does not avoid the event; it only takes the date out of your hands. The fourth case below is a whole family of models moved that way. In January 2024 OpenAI introduced gpt-4-turbo-preview, a name that always points at its latest preview model, on the same day it released a snapshot meant to reduce cases of “laziness”, where the previous one did not finish a task. Whoever called the alias got the new behaviour the day it shipped, wanted or not. A pinned snapshot turns that into a change you schedule and test.

The drill has four moves. As soon as the notice arrives, while the old pin still works, run the full suite against the replacement snapshot with today’s prompt. Expect a long list of flips: Sclar’s finding that the best prompt format for one model barely predicts the best for another means a model change is also a prompt change. Write the next prompt version for the new model and run it again until the protected set holds. Then switch on a day you chose, keep the old pin as the way back until it retires, and record the new baseline.

Four changes, and what a per-case diff would have shown

Each of these was a change to a system that had been working, and in each the regression reached users before any team’s suite reported it. In two of the four, evaluations had been run and had passed.

Where it degrades

The honest limit is that a per-case diff compares the cases you have. A regression on a kind of question nobody put in the set passes every gate on this page; the sycophancy update broke a behaviour no case asked about. The suite tells you that a change did not break what you already knew to check. Error analysis on fresh production conversations, described on the AI evaluation page, is what adds the next case, and nothing here replaces it.

• Judging a change on the aggregate — the statistic in which a fix cancels a break.
• One run per case, so a sampling accident reads as a regression, or hides one.
• An index rebuild, a tool change or a new judge landing in the same week as the prompt edit, and all of it credited to the prompt.
• Editing the production prompt in a console: a deployment with no diff, no review and no way back.
• Re-running only the results nobody liked.
• A protected set that shrinks whenever a case becomes inconvenient.
• Lists of flips nobody reads, so a judge’s mistake is counted as a fix.
• A model alias in production, so the provider decides when your feature changes.
• Discovering a retirement date in the week it arrives.
• A suite that runs only on your changes, blind to the ones you did not make.

When to use it

Related pages

• QAAI evaluationAI evaluation measures whether a model’s outputs are good enough to ship, with a fixed set of cases, a grader for each and numbers read with their uncertainty. The page opens with sixteen things a team will want to grade — the cheapest grader that settles each, and what fools it — then covers error analysis, building the set, the ladder of graders, model judges and how to trust one, reading the numbers, and four published measurements that misled.
• QATesting LLM productsTesting an LLM product means testing a feature whose model answers the same question differently on two runs, while most of its checks remain exact assertions. The page opens with twenty failures an order assistant ships with — the check that catches each, and where it runs — then covers the missing test oracle, the deterministic layer and its unhappy paths, properties for answers with no expected value, the 2026 OWASP list as a test plan, who owns what, and five releases that went wrong.
• QARegression testingRegression testing is re-running tests after a change to catch anything that used to work and has stopped, in parts of the system nobody meant to touch. The page opens with sixteen cases from one shop’s suite — why each is there, the level that proves it and when it runs — then covers what earns a case and what removes one, which subset runs when, flaky tests by the numbers, the cost nobody budgets for, and four regressions where the case was missing.