LLMOps

The production checklist

Here is the whole checklist first, because it is what a readiness review needs and because almost none of it is expensive if it is done before launch. It is numbered so a line can be cited: "we do not have 3.4" is a ticket, where "observability needs work" is a mood. Groups 1 and 3 are the ones to do first — without a pinned model an evaluation set is measuring something that can change between two runs, and without a trace id a complaint is an anecdote.

1. 1.Versioning

   1. 1.1The model version is pinned explicitly. No alias that tracks whatever the provider released last.
   2. 1.2The prompt is in version control beside the code that sends it, and carries a version id.
   3. 1.3The retrieval index has a version, and that version is part of what is deployed.
   4. 1.4Changing any of the four is a deployment: reviewed, announced, and with the previous combination still deployable.

2. 2.Evaluation

   1. 2.1An evaluation set of thirty to fifty cases exists, weighted towards the inputs that break things.
   2. 2.2It runs on every change to the prompt, the model or the retrieval, and reports rates rather than pass or fail.
   3. 2.3Deterministic checks are used wherever the task allows one — labels, schemas, arithmetic, citation existence.
   4. 2.4Any judge model is calibrated against human grades before it is trusted, and pinned like any other model.
   5. 2.5A held-back portion of the set is looked at only before a release, so tuning cannot quietly consume it.

3. 3.Tracing

   1. 3.1Every call records the prompt version, the rendered prompt, and the model and its version.
   2. 3.2Every call records the retrieved chunk ids and every tool calltool callthe model asking code to runA model response that, instead of text, names one of the functions you described to it and supplies arguments; your code runs it and hands the result back. The arguments are untrusted input and are validated like any request body — a model that has read an injected instruction produces a well-formed, entirely wrong call.Full article →All terms with its arguments and its result.
   3. 3.3Every call records tokens in and out, latency, and the final answer as shown to the user.
   4. 3.4Every answer shown to a user carries a trace id that the interface can hand back.

4. 4.Guardrails

   1. 4.1The response is validated against a schema before any code acts on it.
   2. 4.2Citations are checked for existence and for actually supporting the claim before the answer is shown.
   3. 4.3There is a defined behaviour for when the provider is unavailable, rate-limited or slow.
   4. 4.4Timeouts and retries with backoffbackoffwaiting longer between retriesWaiting before retrying a failed call, and waiting longer after each failure — for example 1, 2, 4 and 8 seconds — usually with a random jitter added so that many clients do not retry at the same moment. Without it, retries arrive exactly when the struggling service can least afford them.All terms are on every call, and batch traffic cannot starve the interactive path.

5. 5.Dashboards from week one

   1. 5.1Tokens per request, because prompts grow silently and this line becomes the invoice.
   2. 5.2Latency at the ninety-fifth percentile, since output length dominates it and the tail is what users feel.
   3. 5.3The refusal and fallback rate, which climbs before anything else does when a prompt or a model shifts.
   4. 5.4The rate of answers rejected by your own guardrails — the closest thing to a real-time quality signal.

6. 6.The improvement loop

   1. 6.1Feedback from users carries the trace id, so a rating opens directly onto what caused it.
   2. 6.2Somebody reviews the week’s negative ratings and classifies each one: retrieval, prompt, model, or fine.
   3. 6.3Real failures are promoted into the evaluation set, so it gets more valuable as the system gets more used.
   4. 6.4One named person owns the quality number and has the authority to block a release on it.

What actually changes

A normal service is verified by assertion: send this, expect that, and a difference is a bug. An LLM feature cannot be verified that way, because the same request legitimately produces different text on two consecutive calls. Every practice that rests on exact comparison therefore has to be replaced rather than adapted — and the teams that skip this step do not notice, because their pipeline stays green while quality drifts underneath it.

The deployed unit is four things

Everything else on this page rests on this section, which is why checklist group 1 comes first. Four things decide how the feature behaves and only one of them is in your build. The prompt can be edited in a console. The index can be rebuilt by a nightly job. And the model can be upgraded by the provider, under an alias that always points at the latest release — a dependency that upgrades itself in production, with no diff, no review and no previous version to return to. It is the one arrangement no other part of your stack would tolerate for a week.

Two capabilities make the rest of the checklist practical and are worth building in the first week. A way to change the prompt version without a code deploy, so a bad prompt is a thirty-second fix rather than a release — with the version still recorded and reviewable, which is the difference between this and editing in a console. And an idempotentidempotentsafe to repeatAn operation that has the same effect whether it runs once or five times. It is what makes a retry safe — and since any network call may be retried without you knowing, the alternative is charging a card twice.All terms way to rebuild the retrieval index, so an index problem is repaired by rerunning a job rather than by restoring a backup. Both sound like conveniences until the first incident, at which point they are the difference between minutes and an afternoon.

Evaluation is the test suite

An evaluation set is a fixed collection of inputs with a way to judge each output, run on every change to the prompt, the model or the retrieval, and reported as rates rather than pass or fail. Thirty to fifty cases is a real starting point — the value is not in coverage but in the cases being the ones that actually go wrong. Grow it from production: every complaint, every incident and every surprising answer becomes a case, which turns the set into an asset that gets more valuable exactly as the system gets more used.

Judging cheaply matters more than judging perfectly, so use the strongest check the task allows before reaching for a model. A label can be compared exactly. A schema can be validated. A cited passage can be checked for existence in the source and for actually containing the claim. A number in the answer can be recomputed. These deterministic checks are fast, free and unambiguous, and they cover more of a real evaluation set than teams expect — often the majority of it.

Calibrate any judge before relying on it — checklist line 2.4, and the diagram above is the procedure. Have people grade fifty answers, run the judge on the same fifty, and look at where the two disagree. If they agree often enough, you have a cheap instrument you can run on every change. If they do not, you have learned that the criteria are ambiguous — which is worth knowing anyway, because an ambiguous criterion is also a prompt the team cannot agree on.

What production needs to record

A single log line saying the request succeeded is useless here, because the interesting failure is a call that returned two hundred and said something wrong. What you need is a trace of the whole interaction: the prompt version and the rendered prompt, the model and its version, the retrieved chunk ids, the raw response, every tool call with its arguments and result, token counts in and out, latency, and the final answer. That record is what turns "the assistant told a customer something incorrect on Tuesday" from an unanswerable complaint into a reproducible case.

One caution that belongs beside that list, because the trace is now the most sensitive store you have: it contains, by construction, every document that was retrieved and every question a user asked. Apply the retention policy, the access control and the redaction rules you would apply to any other copy of that material, decide how long a trace lives before the answer to "can we reproduce a complaint from March" becomes no, and make sure the people who need to review negatives can reach it without being handed the whole corpus.

Cost and latency under real traffic

Cost control has four levers and they are worth pulling in order, because the first two are nearly free and the last two are architecture. The table is the order.

Latency needs one thing said plainly, because it catches teams who treat the model like a database. Time to the first token is roughly fixed and everything after it is proportional to how much the model writes, so a slow response is almost always a long one and no amount of infrastructure tuning will change that. Stream the answer, ask for shorter output when you want lower latency, and watch the ninety-fifth percentile rather than the mean — the mean is dominated by the short answers nobody was waiting for.

Five incidents, and which line would have caught each

These are operations failures rather than model failures, which is why they belong here rather than on the fundamentals page. In each one the model did something unremarkable and the surrounding system had no way to notice, no way to reproduce it, or no way to get back.

Where it degrades

The honest limit of everything on this page is that it measures what somebody thought to put in the set. An evaluation suite is a sample of the world, assembled by people who have already decided what matters, and a system can drift in a direction nobody wrote a case for while every number stays green. That is not an argument against measuring — it is the argument for keeping a person in the loop who is allowed to say that something feels wrong and be taken seriously.

• An unpinned model alias, which upgrades itself in production with no diff and no rollback.
• Prompts edited in a console, so the deployed text is not in version control and cannot be reviewed.
• Logging only that the call succeeded, leaving a wrong answer with nothing to reproduce it from.
• An evaluation set built from cases that already pass, which reports a high score and detects nothing.
• A judge model used as the sole release gate, rewarding confident formatting over correctness.
• Tuning the prompt against the same cases you measure with, until the score means nothing.
• A trace store nobody applied a retention or access policy to, holding every document and question in the product.
• A feedback button whose data nobody routes anywhere, and a quality number nobody owns.
• Treating a green dashboard as permission to stop listening to the people who use the feature daily.

When to use it

Related pages

• DevelopmentPrompt EngineeringPrompt engineering is the work of writing and maintaining the instruction a language model acts on, as a specification rather than an incantation. The page opens with a twenty-four-line review checklist you can run against your own prompt today, then the diagnosis that decides whether the prompt is even the right thing to edit, what a prompt is made of, the techniques that genuinely move the output, the evaluation set without which none of it is measurable, injection as an architectural boundary, and five documented failures.
• DevelopmentAI AgentsAn AI agent is a loop in which the model decides what to do next: it calls a tool, reads the result, and decides again, until it judges the task done. That single design choice — control flow moved from your code into the model — buys the ability to handle tasks you cannot specify in advance, and costs you determinism, a predictable bill and a bounded blast radius. The page opens with the decision of whether you need one at all, then covers the loop and the arithmetic that governs it, the tool surface, the failure modes that only appear in a loop, the guardrails and the nineteen-line readiness checklist, how to evaluate a trajectory, and five documented runs.
• DevelopmentRAG ArchitectureRetrieval-augmented generation (RAG) answers questions about data a model was never trained on: find the relevant passages first, then put them in the prompt. Requiring the answer to come from those passages makes RAG a search system with a language model on the end, which is why almost every wrong answer is a retrieval failure rather than a generation one. The page opens with the twenty-four-line build checklist, then nine symptoms and where each is actually fixed, the two pipelines, how to measure retrieval on its own, the permission problem that turns a demo into an incident, the alternatives, and five documented failures.