RAG Architecture
Retrieval-augmented generation (RAG) is how a model answers questions about data it was never trained on: find the relevant passages first, put them in the prompt, and require the answer to come from them. That makes RAG a search system with a language model on the end — and it means almost every wrong answer is a retrieval failure, not a generation one. This page covers both pipelines, how to measure retrieval separately, and the permission problem that turns a demo into an incident.
- Two pipelines
- Indexing and querying
- Most failures
- Retrieval, not generation
- Access control
- Belongs in the index
Two pipelines, not one feature
RAG is drawn as a single arrow and built as two independent systems that run on completely different schedules. The indexing pipeline runs when documents change: it loads them, splits them into chunks, turns each chunk into a vector, and writes it to a store along with its metadata. The query pipeline runs on every user question: it embeds the question, retrieves candidate chunks, narrows them down, assembles a prompt and calls the model. Treating them as one thing is why the second week of a RAG project is usually spent discovering that reindexing was never designed.
An embedding is the mechanism that makes the first pipeline useful. A model converts a piece of text into a list of numbers positioned so that texts with similar meaning land near each other, which is why a question about "cancelling a subscription" can find a passage that says "terminating your plan" with no word in common. The same embedding model must be used for documents and for questions, and changing it means rebuilding the entire index — vectors from two different models are not comparable, and mixing them produces retrieval that is quietly, unfixably wrong.
Chunking is the decision with the most consequences and the least attention. Chunks that are too large bring irrelevant text along with the answer and dilute what the model is reading; chunks that are too small win the similarity match and then turn out to be missing the sentence that made them meaningful. What works better than any fixed size is splitting on the document structure — a heading and the section beneath it, a table with its caption, a function with its docstring — and carrying the document title and section path into each chunk, so a fragment that starts "This does not apply to trial accounts" still says what "this" was.
Measure retrieval before you judge the answer
When a RAG system answers badly, the instinct is to work on the prompt. It is almost always the wrong end. If the passage containing the answer never reached the prompt, no instruction can recover it — the model was asked to answer from material that did not contain the answer, and it did the only thing it can do. So the first diagnostic is mechanical: look at what was retrieved. If the right passage is not there, everything downstream is irrelevant.
That makes retrieval something you evaluate on its own, with its own numbers, before any answer is graded. Build a set of real questions and mark, for each, which chunks actually contain the answer. Then measure how often the right chunk appears anywhere in the top k results at all — recall is the number that matters, because a chunk that was not retrieved cannot be rescued later, whereas an extra irrelevant chunk usually only costs tokens. Track it as a number that moves when you change chunking, the embedding model or the value of k, and you have converted an argument about quality into an experiment.
| Symptom | Usual cause | Where to fix it |
|---|---|---|
| Confident answer, invented detail | The passage was never retrieved, and the prompt gave no way to say so. | Retrieval, plus an explicit "answer only from the context, otherwise say you do not know". |
| Right document, wrong sentence | Chunks are too large, so the relevant line is buried among paragraphs of noise. | Smaller structural chunks, and a reranking pass over the candidates. |
| Exact identifiers never found | Vector similarity blurs codes, SKUs and names into their neighbours. | Hybrid search — run keyword retrieval alongside and merge the two result lists. |
| Stale or withdrawn content quoted | The index was built once and never reconciled with deletions. | The indexing pipeline: incremental updates keyed by document id, deletions included. |
| Good answers, unusable latency | Too many chunks retrieved and every one of them sent to the model. | Retrieve widely, rerank, then pass only the few best into the prompt. |
Reranking is the step that resolves most of the middle of that table. Retrieval is tuned to be fast and generous, so it returns candidates that are roughly relevant; a reranker is a smaller model that reads the question and each candidate together and scores how well it actually answers. Retrieving thirty candidates and passing the best four to the model is consistently better than retrieving four directly, because the cheap wide net and the careful narrow judgement are different jobs done by different tools.
RAG, long context or fine-tuning
Context windows large enough to hold entire handbooks made the obvious question worth asking: why retrieve at all, when you can paste everything in? For a genuinely small and stable corpus — a product manual, a policy document, a single codebase file — pasting it in is the right answer, and building a retrieval pipeline for it is over-engineering. The reasons that break down are quantitative and they arrive quickly: cost scales with every token on every request, latency scales with the prompt, attention across a very long prompt is uneven, and no window holds a corporate wiki. Retrieval is what makes the input size independent of the corpus size.
Reach for RAG
- The knowledge changes — daily prices, tickets, documents edited by people who do not deploy code.
- Answers must cite a source, and a user has to be able to check the claim.
- Different users may see different subsets of the same corpus.
- The corpus is far larger than any prompt, and most of it is irrelevant to any one question.
Reach for fine-tuning
- The problem is behaviour and format, not facts — a house style, a taxonomy, a rigid output shape.
- The same instructions repeat on every call and dwarf the actual input.
- You have thousands of labelled examples and a held-out set to prove it worked.
- Volume is high enough that a shorter prompt pays back the training and the maintenance.
The two are not rivals and mature systems usually run both: a tuned model that knows the format and the tone, answering from passages retrieved at query time. What does not work is using fine-tuning to teach facts. Training on a document set does not create a lookup table — the facts blur into the weights, come back subtly altered, cannot be cited, and cannot be corrected without retraining. Every fact you might need to update, cite or restrict belongs in retrieval.
How this shows up in real delivery
The failure that ends careers is not a wrong answer, it is a correct one. A RAG index built by crawling everything the crawler could reach will happily answer a question with a passage from a salary review, a legal draft or another customer’s ticket — the retrieval worked exactly as designed. Permissions have to be part of the index: store the access rules with each chunk, and filter by the current user before the similarity search rather than after, so a document the user cannot see is never a candidate. Filtering the results afterwards leaks through relevance scores, counts and latency, and re-running an ingest over a source whose permissions changed is a routine operational task, not an afterthought.
Freshness is the other operational reality. The index is a copy, so it is wrong from the moment a document changes until the moment it is reindexed, and how wrong is a product decision somebody should make consciously. Drive it by change events where the source can emit them and fall back to a scheduled sweep where it cannot; key chunks by document id so an update replaces rather than duplicates; and handle deletions explicitly, because an answer quoting a policy that was withdrawn last month is worse than no answer at all.
On storage: a dedicated vector database is not the automatic answer. Postgres with a vector extension, or the vector support in a search engine you already run, handles millions of chunks and gives you one system to operate, one backup, one set of transactional guarantees, and metadata filtering that actually works. That matters more than raw search speed for most products, and hybrid search is easier when both the keyword index and the vectors live in the same place. Reach for a specialised store when scale, index build time or a specific algorithm genuinely demands it — and know which of those you are buying.
Where it degrades
- Debugging the prompt when the right passage never reached it, which is most of the time.
- Fixed-size chunking that cuts through tables, code and sentences and destroys the meaning it was indexing.
- An index built once by hand, with no incremental update path and no story for deletions.
- Access control applied to the results instead of the query, which leaks through scores, counts and timing.
- Changing the embedding model without rebuilding the index, leaving two incomparable vector spaces in one store.
- Vector-only retrieval on a corpus full of codes, identifiers and names that keyword search would find exactly.
- No citations, so no user can verify an answer and no engineer can reproduce a complaint.
When to use it
Use it when
- Answering from a corpus that changes on its own schedule — documentation, policies, tickets, catalogues.
- Anywhere the answer must be traceable to a source a user can open and check.
- Corpora too large for any context window, where only a fraction is relevant to a given question.
- Products where different users are entitled to different documents, and the boundary must be enforced per query.
Avoid it when
- A corpus small and stable enough to paste into the prompt, where retrieval adds a pipeline and no accuracy.
- Problems of style, tone or output format, which retrieval cannot touch — that is prompting or fine-tuning.
- Questions needing aggregation over a whole dataset — counts, sums, trends — which a query against the database answers exactly.
- Sources whose permissions you cannot represent per chunk, where a correct retrieval is itself the incident.
Found this useful?
Share it with someone who is working on the same problem.