LLM Fundamentals

One operation, repeated

The model does not read words. Text is first cut into tokens — fragments of roughly three or four characters in English, and often a single character or two in Ukrainian, which is why the same sentence costs noticeably more tokens in Ukrainian than in English. Every token has a number, and the model only ever sees that sequence of numbers. Ask it how many letters are in a word and it is guessing about something it cannot see, which is the real reason such questions go wrong.

Given that sequence, the model produces one thing: a score for every token in its vocabulary, saying how well each would continue the text. It does not produce an answer, a sentence or an opinion — it produces a distribution over perhaps a hundred thousand candidates. A sampler then picks one, appends it to the input, and the whole calculation runs again for the next token. A four-hundred-word answer is that loop run several hundred times, and this is why output is slower and more expensive than input: the input is processed in one pass, while every output token is a separate trip through the model.

The sampler is where the settings you can actually change live, and there are fewer of them than the tooling suggests. They shape how the distribution is turned into a choice — they cannot add knowledge, and they cannot make the model more careful. Most teams touch two of them and leave the rest alone.

The context window is the whole memory

The context window is the maximum number of tokens the model can have in front of it on one call, and it holds everything at once: the system instructions, the conversation so far, any documents you pasted in, the tool definitions, and the answer being generated. It is not storage. Between two calls the model retains nothing whatsoever — the API is stateless, exactly like HTTP.

A large window is not the same as attention to everything inside it. Models reliably use what is near the beginning and near the end of the prompt, and use the middle less well — a long document dropped in the centre of a request is the classic way to get an answer that ignores half of it. Put the instruction and the actual question at the end, closest to where generation starts, and keep the padding out entirely. Filling a million-token window because it is available is a cost and a latency decision as much as a quality one.

This is also where prompt caching changes the economics, and it is the one optimisation worth understanding early. Providers can cache the processed form of a prefix of your request, so a repeated prefix is billed at a fraction of the normal input price and processed faster. The catch is that it is a prefix match: one changed byte anywhere before the end of the cached region invalidates all of it. Practically, that means putting the stable material first — system instructions, tool definitions, the long document — and everything volatile last. A timestamp or a request id at the top of the system prompt quietly destroys the cache on every single call, and nothing in the response tells you so except the usage counters.

The limits that are structural, not bugs

A model that always produces the most plausible continuation will produce a plausible continuation even when there is no true one available. That is what a hallucination is: not a malfunction, but the same mechanism working on a question the weights have nothing solid for. It has no separate state for "I do not know", so the confident tone of a wrong answer is identical to the confident tone of a right one, and confidence carries no information you can use.

Two more limits follow from how a model is made. Its weights were frozen at a training cutoff, so anything later — a release, a price change, an incident, your own documentation from last week — simply is not in there, and asking about it produces the plausible-continuation behaviour above rather than an admission. And the same prompt is not guaranteed to give the same answer twice: sampling is random by construction, and even at the lowest temperature setting, batching and hardware differences on the provider side keep exact reproducibility off the table. Any test that asserts on an exact response string will eventually fail for no reason connected to your change.

How this shows up in real delivery

Model choice is a range, not a decision made once. Providers ship a capable tier and a cheap fast tier, and the useful habit is to prove the task is possible on the strongest model available, then try to walk down the range with a fixed set of test cases in hand. Without that set you are not choosing a model, you are guessing at one — and the cheaper model will look fine on the three examples you happen to try.

The cost line on an LLM feature is almost always tokens per request multiplied by requests, and the term that grows unnoticed is the first one. Prompts accumulate: an instruction added after each incident, a longer example, three more retrieved documents, and the conversation history under all of it. Log the token counts of every call from the first day, alongside latency — it is the only way to see the growth, and it turns a vague "the AI feature got expensive" into a specific prompt you can trim.

Latency behaves unlike any other dependency you have. Time to the first token is roughly fixed, and everything after it is proportional to how much the model writes — so a request that produces a long answer is slow because it is long, and no amount of infrastructure tuning changes that. Two things follow: stream the response so the user sees progress instead of a spinner, and ask for shorter output when you actually want lower latency, because that is the variable that moves it.

Where it degrades

• Treating the model as a knowledge base about your own product, instead of putting your documents into the prompt.
• Tests that assert an exact response string, which fail on a change of sampling rather than a change of behaviour.
• A timestamp or request id at the top of the system prompt, which invalidates the prompt cache on every call.
• Raising the temperature to make answers "smarter", when it only makes unlikely tokens more likely.
• Letting a chat grow until a library silently truncates the beginning, taking the system instructions with it.
• Filling a huge context window because it exists, paying for tokens that make the answer worse rather than better.
• Shipping without token and latency metrics, so the first sign of a problem is the invoice.

When to use it

Related pages

• DevelopmentPrompt EngineeringPrompt engineering is the work of writing the instruction a language model acts on. A prompt is a specification, not an incantation. The phrases that get traded as tricks work only because they add information the model did not otherwise have, and the ones that do not add information do nothing at all. This page covers what a prompt is made of, which techniques genuinely change the output, how to tell whether a change helped, and why untrusted text in a prompt is a security boundary rather than a wording problem.
• DevelopmentRAG ArchitectureRetrieval-augmented generation (RAG) is how a model answers questions about data it was never trained on: find the relevant passages first, put them in the prompt, and require the answer to come from them. That makes RAG a search system with a language model on the end — and it means almost every wrong answer is a retrieval failure, not a generation one. This page covers both pipelines, how to measure retrieval separately, and the permission problem that turns a demo into an incident.
• DevelopmentLLMOpsLLMOps is the practice of running a feature built on a language model in production, and it exists because everything that makes software releasable — a test that fails, a diff that explains a change, a rollback that restores yesterday — assumes the same input gives the same output. LLM features break that assumption, so the practices have to be rebuilt around it. This page covers what replaces the test suite, what the deployable artefact actually is, what to record on every call, and how cost and latency behave once real traffic arrives.