LLM Fundamentals

Twelve things that look like bugs and are not

Almost everyone arrives at this subject holding a specific complaint rather than a general question: the model did something that looked broken. Here is the whole list first, because it is the part you can use today, and because each row has an exact mechanical cause rather than a mood. The right-hand column matters more than it looks: eleven of these twelve are fixed somewhere other than the prompt, and reaching for the wrong correction is the most expensive habit on an AI project. Every mechanism named here is explained in full further down, and the row numbers are referred to from those sections.

One operation, repeated

The model does not read words. Text is first cut into tokens — fragments of roughly three or four characters in English — and every token has a number, so what the model actually receives is a sequence of numbers. That is the whole of row 1: ask it how many letters are in a word and it is guessing about something it has never seen, the way you would guess at the number of grooves on a record you were handed in a sealed sleeve.

Given that sequence, the model produces one thing: a score for every token in its vocabulary, saying how well each would continue the text. It does not produce an answer, a sentence or an opinion — it produces a distribution over perhaps a hundred thousand candidates. A sampler then picks one, appends it to the input, and the whole calculation runs again for the next token. A four-hundred-word answer is that loop run several hundred times, and this is why output is slower and more expensive than input: the input is processed in one pass, while every output token is a separate trip through the model.

The sampler is where the settings you can actually change live, and there are fewer of them than the tooling suggests. They shape how the distribution is turned into a choice — they cannot add knowledge, and they cannot make the model more careful. It is also where row 4 comes from: the same prompt can give different answers because a pick from a distribution is a pick, and even at the lowest temperature, batching and hardware differences on the provider side keep exact reproducibility off the table.

Why the same sentence costs more in Ukrainian

A tokeniser has a fixed vocabulary, learned once from a training corpus, and it keeps whole what it saw often. That corpus is overwhelmingly English, so English words survive nearly intact while Cyrillic is cut into fragments of one or two characters. The consequence is row 10, and it is arithmetic rather than opinion: the same meaning, expressed in Ukrainian, arrives as roughly two to three times as many tokens.

Two practical consequences follow, and a Ukrainian-language product meets both. The first is the invoice: identical work costs more, on every request, forever. The second is quieter and matters more — the context window is counted in tokens, so the Ukrainian version of a feature runs out of room sooner than the English one. A chat history, a retrieved document set or a few-shot example block that fits comfortably in English can be the thing that pushes the Ukrainian version past the limit, and the symptom will look like row 2 rather than like a budgeting problem.

The context window is the whole memory

The context window is the maximum number of tokens the model can have in front of it on one call, and it holds everything at once: the system instructions, the conversation so far, any documents you pasted in, the tool definitions, and the answer being generated. It is not storage. Between two calls the model retains nothing whatsoever — the APIAPIApplication Programming Interface — contract between programsThe rules one program follows to call another: which requests it may send, in what shape, and what comes back. To the caller it is a contract — everything behind it stays a black box, which is exactly what lets the other side be rewritten without breaking you.Full article →All terms is stateless, exactly like HTTPHTTPHyperText Transfer Protocol — the protocol of the webThe protocol browsers and servers talk in: the client sends a method and an address, the server answers with a status code and a body. It keeps no memory between requests — every request stands alone, which is why anything that must be remembered is carried explicitly.All terms.

A large window is not the same as attention to everything inside it. Models reliably use what is near the beginning and near the end of the prompt, and use the middle less well — that is row 5, and a long document dropped in the centre of a request is the classic way to get an answer that ignores half of it. Put the instruction and the actual question at the end, closest to where generation starts, and keep the padding out entirely. Filling a million-token window because it is available is a cost and a latency decision as much as a quality one.

This is also where prompt caching changes the economics, and it is the one optimisation worth understanding early. Providers can cache the processed form of a prefix of your request, so a repeated prefix is billed at a fraction of the normal input price and processed faster. The catch is that it is a prefix match: one changed byte anywhere before the end of the cached region invalidates all of it. Practically, that means putting the stable material first — system instructions, tool definitions, the long document — and everything volatile last. A timestamp or a request id at the top of the system promptsystem promptthe standing instructionsThe part of a prompt carrying the role and the rules, sent on every call and normally not shown to the user. It is not a secret: it sits in the same window as everything else and has been extracted from production systems by asking, so anything genuinely confidential belongs outside it.Full article →All terms quietly destroys the cache on every single call, and nothing in the response tells you so except the usage counters.

The limits that are structural, not bugs

A model that always produces the most plausible continuation will produce a plausible continuation even when there is no true one available. That is what a hallucination is, and it is row 3: not a malfunction, but the same mechanism working on a question the weights have nothing solid for. It has no separate state for "I do not know", so the confident tone of a wrong answer is identical to the confident tone of a right one, and confidence carries no information you can use.

Two more limits follow from how a model is made. Its weights were frozen at a training cutoff, so anything later — a release, a price change, an incident, your own documentation from last week — simply is not in there, and asking about it produces the plausible-continuation behaviour above rather than an admission. That is row 8. And the refusals are the same kind of thing from the other direction: a refusal is learned behaviour over text rather than a policy engine, which is why row 11 happens and why any rule that genuinely must hold belongs in your own code.

Five failures and what was missing from each

None of these went wrong because the model malfunctioned. In every one the model did exactly what the mechanism above says it does, and the missing piece was a control somebody had assumed was unnecessary. They are worth knowing by name because each one is now the cheapest possible way to make an argument for that control in a planning meeting.

What it costs once real traffic arrives

Model choice is a range, not a decision made once. Providers ship a capable tier and a cheap fast tier, and the useful habit is to prove the task is possible on the strongest model available, then try to walk down the range with a fixed set of test cases in hand. Without that set you are not choosing a model, you are guessing at one — and the cheaper model will look fine on the three examples you happen to try.

The cost line on an LLM feature is almost always tokens per request multiplied by requests, and the term that grows unnoticed is the first one. Prompts accumulate: an instruction added after each incident, a longer example, three more retrieved documents, and the conversation history under all of it. Log the token counts of every call from the first day, alongside latency — it is the only way to see the growth, and it turns a vague "the AI feature got expensive" into a specific prompt you can trim.

Latency behaves unlike any other dependency you have. Time to the first token is roughly fixed, and everything after it is proportional to how much the model writes — so a request that produces a long answer is slow because it is long, and no amount of infrastructure tuning changes that. Two things follow: stream the response so the user sees progress instead of a spinner, and ask for shorter output when you actually want lower latency, because that is the variable that moves it.

Treat the provider as what it is: a third-party dependency with rate limits and an availability record you do not control. Every call needs a timeout and a retry with backoffbackoffwaiting longer between retriesWaiting before retrying a failed call, and waiting longer after each failure — for example 1, 2, 4 and 8 seconds — usually with a random jitter added so that many clients do not retry at the same moment. Without it, retries arrive exactly when the struggling service can least afford them.All terms, because a hung request holds a connection far longer than any database would. And decide in advance what the feature does when the model is unavailable, since "the page hangs" is a decision too, just not one anybody made.

Where it degrades

Everything above describes a component that generates plausible text and does nothing else. The honest limit of that component is easy to state and hard to accept: there is no configuration, no prompt and no model version that makes a generated answer self-verifying. Every arrangement below is an attempt to skip that fact, and each one fails in a way the twelve rows already predicted.

• Treating the model as a knowledge base about your own product, instead of putting your documents into the prompt.
• Tests that assert an exact response string, which fail on a change of sampling rather than a change of behaviour.
• A timestamp or request id at the top of the system prompt, which invalidates the prompt cache on every call.
• Raising the temperature to make answers "smarter", when it only makes unlikely tokens more likely.
• Letting a chat grow until a library silently truncates the beginning, taking the system instructions with it.
• Filling a huge context window because it exists, paying for tokens that make the answer worse rather than better.
• Sizing a Ukrainian feature on English token counts, so the window and the budget are both wrong by a factor of two.
• Treating a refusal as a security control, when it is learned behaviour that a rephrase can walk around.
• Shipping without token and latency metrics, so the first sign of a problem is the invoice.

When to use it

Related pages

• DevelopmentPrompt EngineeringPrompt engineering is the work of writing and maintaining the instruction a language model acts on, as a specification rather than an incantation. The page opens with a twenty-four-line review checklist you can run against your own prompt today, then the diagnosis that decides whether the prompt is even the right thing to edit, what a prompt is made of, the techniques that genuinely move the output, the evaluation set without which none of it is measurable, injection as an architectural boundary, and five documented failures.
• DevelopmentRAG ArchitectureRetrieval-augmented generation (RAG) answers questions about data a model was never trained on: find the relevant passages first, then put them in the prompt. Requiring the answer to come from those passages makes RAG a search system with a language model on the end, which is why almost every wrong answer is a retrieval failure rather than a generation one. The page opens with the twenty-four-line build checklist, then nine symptoms and where each is actually fixed, the two pipelines, how to measure retrieval on its own, the permission problem that turns a demo into an incident, the alternatives, and five documented failures.
• DevelopmentLLMOpsLLMOps is the practice of running a feature built on a language model in production, where the same input does not reliably produce the same output. It exists because everything that makes software releasable — a test that fails, a diff that explains a change, a rollback that restores yesterday — assumes that it does. The page opens with the twenty-four-line production checklist, then covers what each practice is replaced by, why the deployed unit is four things rather than one, evaluation as the test suite, what a trace has to contain, how cost and latency behave under real traffic, and five documented incidents.