LLMOps

What actually changes

A normal service is verified by assertion: send this, expect that, and a difference is a bug. An LLM feature cannot be verified that way, because the same request legitimately produces different text on two consecutive calls. Every practice that rests on exact comparison therefore has to be replaced rather than adapted — and the teams that skip this step do not notice, because their pipeline stays green while quality drifts underneath it.

Evaluation is the test suite

An evaluation set is a fixed collection of inputs with a way to judge each output, run on every change to the prompt, the model or the retrieval, and reported as rates rather than pass or fail. Thirty to fifty cases is a real starting point — the value is not in coverage but in the cases being the ones that actually go wrong. Grow it from production: every complaint, every incident and every surprising answer becomes a case, which turns the set into an asset that gets more valuable exactly as the system gets more used.

Judging cheaply matters more than judging perfectly, so use the strongest check the task allows before reaching for a model. A label can be compared exactly. A schema can be validated. A cited passage can be checked for existence in the source and for actually containing the claim. A number in the answer can be recomputed. These deterministic checks are fast, free and unambiguous, and they cover more of a real evaluation set than teams expect — often the majority of it.

Calibrate any judge before relying on it: have people grade fifty answers, run the judge on the same fifty, and look at where the two disagree. If they agree often enough, you have a cheap instrument you can run on every change. If they do not, you have learned that the criteria are ambiguous — which is worth knowing anyway, because an ambiguous criterion is also a prompt the team cannot agree on.

What production needs to record

A single log line saying the request succeeded is useless here, because the interesting failure is a call that returned two hundred and said something wrong. What you need is a trace of the whole interaction: the prompt version and the rendered prompt, the model and its version, the retrieved chunk ids, the raw response, every tool call with its arguments and result, token counts in and out, latency, and the final answer. That record is what turns "the assistant told a customer something incorrect on Tuesday" from an unanswerable complaint into a reproducible case.

Four signals are worth a dashboard from the first week. Tokens per request, because prompts grow silently and this is the line item that becomes the invoice. Latency at the ninety-fifth percentile, since output length dominates it and the tail is what users feel. The refusal and fallback rate, which climbs before anything else does when a prompt or a model shifts. And the rate of answers rejected by your own guardrails — schema failures, missing citations, blocked content — which is the closest thing to a real-time quality signal you will get.

Roll changes out the way you would any risky dependency: a share of traffic first, the four signals watched, the previous prompt and model pin still deployable. Two things make that possible and are worth building early — a way to change the prompt version without a code deploy, so a bad prompt is a thirty-second fix rather than a release, and an idempotent way to rebuild the retrieval index, so an index problem is repaired by rerunning a job rather than by restoring a backup.

How this shows up in real delivery

Cost control has three levers and they are worth pulling in order. Shorten the input, because it is usually where the tokens are — trimmed history, fewer retrieved chunks, examples that earn their place. Arrange the prompt so the provider cache can hold its stable prefix, which is close to free and often the largest single saving. Then route by difficulty: send the easy majority to a small fast model and reserve the capable one for the cases that need it, with the evaluation set proving where the line sits. Work that nobody is waiting for — reindexing, backfills, bulk classification — belongs in a batch pipeline, which providers price well below interactive calls.

Treat the provider as what it is: a third-party dependency with rate limits and an availability record you do not control. Every call needs a timeout and a retry with backoff, because a hung request holds a connection far longer than any database would. Rate limits are per organisation, so one runaway batch job can starve the interactive path — separate the keys or the queues. And decide in advance what the feature does when the model is unavailable, since "the page hangs" is a decision too, just not one anybody made.

One organisational point, because it decides more outcomes than any tool. An LLM feature has no natural owner: the prompt is written by a product person, the retrieval by a backend engineer, the evaluation set by whoever cared, and the invoice arrives somewhere else entirely. Name one owner for the quality number and give them the authority to block a release on it. Without that, the evaluation set stops being run within a quarter, and the first person to notice quality slipping is a customer.

Where it degrades

• An unpinned model alias, which upgrades itself in production with no diff and no rollback.
• Prompts edited in a console, so the deployed text is not in version control and cannot be reviewed.
• Logging only that the call succeeded, leaving a wrong answer with nothing to reproduce it from.
• An evaluation set built from cases that already pass, which reports a high score and detects nothing.
• A judge model used as the sole release gate, rewarding confident formatting over correctness.
• Tuning the prompt against the same cases you measure with, until the score means nothing.
• A feedback button whose data nobody routes anywhere, and a quality number nobody owns.

When to use it

Related pages

• DevelopmentPrompt EngineeringPrompt engineering is the work of writing the instruction a language model acts on. A prompt is a specification, not an incantation. The phrases that get traded as tricks work only because they add information the model did not otherwise have, and the ones that do not add information do nothing at all. This page covers what a prompt is made of, which techniques genuinely change the output, how to tell whether a change helped, and why untrusted text in a prompt is a security boundary rather than a wording problem.
• DevelopmentAI AgentsAn AI agent is a loop in which the model decides what to do next: it calls a tool, reads the result, and decides again, until it judges the task done. That single design choice — control flow moved from your code into the model — buys the ability to handle tasks you cannot specify in advance, and costs you determinism, a predictable bill and a bounded blast radius. This page covers the loop, the failure modes that only appear in one, and how to tell whether you need an agent at all.
• DevelopmentRAG ArchitectureRetrieval-augmented generation (RAG) is how a model answers questions about data it was never trained on: find the relevant passages first, put them in the prompt, and require the answer to come from them. That makes RAG a search system with a language model on the end — and it means almost every wrong answer is a retrieval failure, not a generation one. This page covers both pipelines, how to measure retrieval separately, and the permission problem that turns a demo into an incident.