RAG Architecture

The RAG build checklist

Here is the whole checklist first, because it is the thing worth having open during a design review rather than after an incident. It is numbered so a line can be cited: "we have not done 4.1" is a decision somebody can schedule, where "the permissions need thinking about" is not. Most RAG prototypes reach a demo having done groups 2 and 5 and none of 1, 4.1, 6 or 7 — and every one of those omissions surfaces later as a class of failure rather than as a bug.

1. 1.Sources and scope

   1. 1.1The corpus is an explicit list of sources, not "everything the crawler could reach".
   2. 1.2Every source has a named owner who can be asked whether a document belongs in there.
   3. 1.3Each source’s access rules can be expressed per document. A source where they cannot is out of scope.

2. 2.Chunking

   1. 2.1Split on document structure — a heading and its section, a table with its caption, a function with its docstring.
   2. 2.2Every chunk carries the document title and the section path, so a fragment still says what it is about.
   3. 2.3Every chunk has a stable id that survives a reindex, so an answer can cite it and a log can name it.
   4. 2.4Tables, code blocks and lists are not cut in half.

3. 3.Embedding and index

   1. 3.1One embedding modelembedding modelturns text into coordinatesA model that converts a piece of text, an image or a product into a list of numbers positioned so that similar things land near each other, which is what lets a search match meaning rather than words. The same one must be used for the documents and for the questions: vectors from two different models are not comparable.Full article →All terms for documents and questions, and its identity is recorded with the index.
   2. 3.2Changing the embedding model means a full rebuild, and there is a written procedure for doing it.
   3. 3.3The index stores access rules and freshness metadata beside each chunk, not in a separate system.

4. 4.Retrieval

   1. 4.1Permission filtering runs before the similarity search, so a forbidden document is never a candidate.
   2. 4.2Keyword search runs alongside vector search and the two result lists are merged.
   3. 4.3Retrieve widely, rerank the candidates, and pass only the best few into the prompt.
   4. 4.4Recallrecallshare of targets foundThe share of the things you were looking for that a system actually found: a recall of 80% means one in five was missed. It is the number that matters when a miss is expensive, and it is traded against precision — how many of the things found were real.All terms on a labelled question set is measured and tracked as a number that moves.

5. 5.Generation

   1. 5.1The prompt says to answer only from the supplied context, and what to say when it does not contain the answer.
   2. 5.2The answer returns the chunk ids it used, as data rather than as prose.
   3. 5.3An answer citing nothing, or citing a chunk whose text does not support it, is refused rather than published.

6. 6.Freshness and deletion

   1. 6.1Updates are incremental and keyed by document id, so an edit replaces a chunk rather than duplicating it.
   2. 6.2Deletions are handled explicitly, and a withdrawn document leaves the index the same day.
   3. 6.3There is a stated, deliberate answer to "how stale is this allowed to be?" and somebody agreed to it.
   4. 6.4Re-indexing a source whose permissions changed is a routine scheduled job, not an incident response.

7. 7.Operations

   1. 7.1The index version is part of what is deployed, alongside the prompt version and the model version.
   2. 7.2What was retrieved is logged with every answer, so a complaint from last week is reproducible.
   3. 7.3A full rebuild is idempotentidempotentsafe to repeatAn operation that has the same effect whether it runs once or five times. It is what makes a retry safe — and since any network call may be retried without you knowing, the alternative is charging a card twice.All terms and can be run on demand without restoring from a backup.

Nine symptoms and where each one is fixed

When a RAG system answers badly, the instinct is to work on the prompt. It is almost always the wrong end. If the passage containing the answer never reached the prompt, no instruction can recover it — the model was asked to answer from material that did not contain the answer, and it did the only thing it can do. So the first diagnostic is mechanical: look at what was retrieved. This table is that lookup, done in advance.

Two pipelines, not one feature

RAG is drawn as a single arrow and built as two independent systems that run on completely different schedules. The indexing pipeline runs when documents change: it loads them, splits them into chunks, turns each chunk into a vector, and writes it to a store along with its metadata. The query pipeline runs on every user question: it embeds the question, retrieves candidate chunks, narrows them down, assembles a prompt and calls the model. Treating them as one thing is why the second week of a RAG project is usually spent discovering that reindexing was never designed — checklist group 6 exists because that discovery is so reliable.

An embedding is the mechanism that makes the first pipeline useful. A model converts a piece of text into a list of numbers positioned so that texts with similar meaning land near each other, which is why a question about "cancelling a subscription" can find a passage that says "terminating your plan" with no word in common. The same embedding model must be used for documents and for questions, and changing it means rebuilding the entire index — vectors from two different models are not comparable, and mixing them produces retrieval that is quietly, unfixably wrong.

Chunking is the decision with the most consequences and the least attention, and the diagram above is the whole argument for checklist group 2. Chunks that are too large bring irrelevant text along with the answer and dilute what the model is reading; chunks that are too small win the similarity match and then turn out to be missing the sentence that made them meaningful. What works better than any fixed size is splitting on the document structure — a heading and the section beneath it, a table with its caption, a function with its docstring — and carrying the document title and section path into each chunk, so a fragment that starts "This does not apply to trial accounts" still says what "this" was.

Measure retrieval before you judge the answer

Retrieval is something you evaluate on its own, with its own numbers, before any answer is graded. Build a set of real questions and mark, for each, which chunks actually contain the answer. Then measure how often the right chunk appears anywhere in the top k results at all — recall is the number that matters, because a chunk that was not retrieved cannot be rescued later, whereas an extra irrelevant chunk usually only costs tokens. Track it as a number that moves when you change chunking, the embedding model or the value of k, and you have converted an argument about quality into an experiment.

Reranking is the step that resolves most of the middle of the symptom table. Retrieval is tuned to be fast and generous, so it returns candidates that are roughly relevant; a reranker is a smaller model that reads the question and each candidate together and scores how well it actually answers. Retrieving thirty candidates and passing the best four to the model is consistently better than retrieving four directly, because the cheap wide net and the careful narrow judgement are different jobs done by different tools — and the funnel above is the shape that follows from taking that seriously.

One practical note on building that labelled question set, because it is the part teams put off. It does not have to be large and it must not be invented: take the questions people actually asked, from support tickets, search logs or the first weeks of the assistant itself, and mark the answering chunk by hand. Fifty of those are worth more than five hundred written by whoever had an afternoon, because the invented ones are phrased the way the person who built the index would phrase them, and that is precisely the query the index already handles.

The correct answer that is the incident

The failure that ends careers is not a wrong answer, it is a correct one. A RAG index built by crawling everything the crawler could reach will happily answer a question with a passage from a salary review, a legal draft or another customer’s ticket — the retrieval worked exactly as designed, the ranking was good, and the answer is accurate. There is no quality metric on this page that would have caught it, which is why permissions are a design decision made before the first document is ingested rather than a filter added when somebody notices.

Store the access rules with each chunk, and filter by the current user before the similarity search rather than after, so a document the user cannot see is never a candidate. Filtering the results afterwards leaks through relevance scores, counts and latency — all three are observable, and a determined user can learn that a document exists, roughly what it is about and when it changed, without ever seeing a line of it. Then treat re-running an ingest over a source whose permissions changed as a routine operational task: a person who left a project keeps their access in the index until somebody reruns the job, and "until somebody reruns the job" is not an access policy.

Freshness is the same problem with the clock running the other way. The index is a copy, so it is wrong from the moment a document changes until the moment it is reindexed, and how wrong is a product decision somebody should make consciously. Drive it by change events where the source can emit them and fall back to a scheduled sweep where it cannot; key chunks by document id so an update replaces rather than duplicates; and handle deletions explicitly, because an answer quoting a policy that was withdrawn last month is worse than no answer at all — it is the Air Canada failure with extra machinery in front of it.

RAG, long context or fine-tuning

Context windows large enough to hold entire handbooks made the obvious question worth asking: why retrieve at all, when you can paste everything in? For a genuinely small and stable corpus — a product manual, a policy document, a single codebase file — pasting it in is the right answer, and building a retrieval pipeline for it is over-engineering. The reasons that break down are quantitative and they arrive quickly: cost scales with every token on every request, latency scales with the prompt, attention across a very long prompt is uneven, and no window holds a corporate wiki. Retrieval is what makes the input size independent of the corpus size.

The two are not rivals and mature systems usually run both: a tuned model that knows the format and the tone, answering from passages retrieved at query time. What does not work is using fine-tuningfine-tuningtraining on your own examplesContinuing a model’s training on examples you supply, so behaviour you keep describing in the prompt becomes what it does by default. It teaches form, tone and judgement well and facts badly: a fact absorbed into weights comes back approximately, cannot be cited and cannot be corrected without training again.Full article →All terms to teach facts. Training on a document set does not create a lookup table — the facts blur into the weights, come back subtly altered, cannot be cited, and cannot be corrected without retraining. Every fact you might need to update, cite or restrict belongs in retrieval, and the three verbs in that sentence are the whole decision rule.

Five systems that retrieved exactly what they were told to

None of these were generation failures and four of them were not even retrieval failures in the technical sense — the systems found what the corpus and the permissions allowed them to find. What went wrong was upstream of the machinery, in decisions about what belonged in the corpus and who was allowed to reach it.

Where it degrades

The honest limit of RAG is that it can only be as good as a search over a corpus somebody curated, and it inherits every problem that corpus has. It cannot tell a joke from a policy, an old draft from the approved version, or a document that is accurate from one that is accurate and confidential. Retrieval quality is a search problem and the remaining risk is an editorial one, which is why the items below are mostly organisational rather than technical.

• Debugging the prompt when the right passage never reached it, which is most of the time.
• Fixed-size chunking that cuts through tables, code and sentences and destroys the meaning it was indexing.
• An index built once by hand, with no incremental update path and no story for deletions.
• Access control applied to the results instead of the query, which leaks through scores, counts and timing.
• A corpus anyone can write to, treated as trusted because it is internal.
• Changing the embedding model without rebuilding the index, leaving two incomparable vector spaces in one store.
• Vector-only retrieval on a corpus full of codes, identifiers and names that keyword search would find exactly.
• No citations, so no user can verify an answer and no engineer can reproduce a complaint.
• A demo corpus that became the production corpus because nobody re-decided what was in it.

When to use it

Related pages

• InfrastructureVector DatabasesA vector database answers "what is most similar to this" over millions of items in milliseconds, and it does it by not looking at most of them. That is the whole trade: every answer is approximate, and how approximate is a number you set rather than a property the product has. This page covers what is actually stored, the index families and what each one costs in memory, why metadata filtering is harder than it looks, what deletes and re-embedding do to an index, and when a column in the database you already run is the better answer.
• DevelopmentFine-tuningFine-tuning continues a model’s training on your own examples, so the behaviour you keep describing in the prompt becomes the behaviour it defaults to. It is excellent at teaching form, tone and judgement calls, and close to useless at teaching facts. The page opens with the eighteen gates worth clearing before a training run and the table that says which rung you actually need, then covers what the methods differ on, why the dataset is the whole project, the held-out comparison that decides whether it was worth doing, what it costs elsewhere in the model, and five documented cases.
• DevelopmentAI AgentsAn AI agent is a loop in which the model decides what to do next: it calls a tool, reads the result, and decides again, until it judges the task done. That single design choice — control flow moved from your code into the model — buys the ability to handle tasks you cannot specify in advance, and costs you determinism, a predictable bill and a bounded blast radius. The page opens with the decision of whether you need one at all, then covers the loop and the arithmetic that governs it, the tool surface, the failure modes that only appear in a loop, the guardrails and the nineteen-line readiness checklist, how to evaluate a trajectory, and five documented runs.