Retrieval-Augmented Generation Explained: Giving AI Access to Your Own Data
The architecture that lets a general model answer questions about your specific documents.
6 min read
Loading…
The architecture that lets a general model answer questions about your specific documents.
Every organisation that tries AI arrives at the same wall within about a fortnight. The model writes beautifully, reasons capably, and knows absolutely nothing about your refund policy, your codebase, or the contract signed last March.
Retrieval-augmented generation — RAG — is the standard answer. The idea is almost embarrassingly simple: before asking the model anything, go and find the relevant passages from your own material and paste them into the prompt. The model then answers from what is in front of it.
The concept takes a minute to grasp. Making it work well takes considerably longer, because nearly all the difficulty lives in the retrieval half, which is the half most tutorials skip.
It is a fair question, and the answer is practical rather than theoretical. Fine-tuning bakes knowledge into weights, which means every update to your documents requires another training run. It is expensive, slow, and the resulting model still cannot tell you where an answer came from.
Retrieval keeps knowledge outside the model, in a store you can update in seconds. Change a policy document, and the next question uses the new version. You also get citations for free, because you know exactly which passages you supplied.
| Fine-tuning | Retrieval (RAG) | |
|---|---|---|
| Updating knowledge | retrain the model | update a document |
| Cost of a change | high, in time and compute | negligible |
| Citations | not available | you know which passages you sent |
| Best at | style, format, behaviour | facts, policies, current data |
| Access control | baked in for everyone | filter per user at query time |
Documents are split into passages small enough to fit several into a prompt, but large enough to stand alone. This is the step that quietly determines everything downstream, and the one people rush.
Split by structure — headings, sections, paragraphs — rather than by a fixed character count, and keep the section heading attached to every chunk. A passage that reads this must be completed within 30 days is useless without the heading that says which process it belongs to.
Each chunk is converted into a vector, using the same kind of embedding model described in our explainer on how language models work. Chunks about similar topics land near each other in that vector space.
The vectors go into a store that can find nearest neighbours quickly. That might be a dedicated vector database, or an extension to a database you already run — Postgres with pgvector handles a surprising amount of production traffic.
The user's question is embedded the same way, the store returns the closest chunks, and those are placed into the prompt with an instruction to answer from them.
A disappointing RAG deployment is almost never a model problem. Run the diagnostic in this order.
This is the most common failure by a wide margin. Check it directly: log the retrieved chunks for a failing question and read them. If the answer is not in there, the model was set up to fail and the prompt is irrelevant.
Semantic similarity is excellent at concepts and mediocre at identifiers. Search for error code E-4021 or a specific product SKU, and pure embedding search will cheerfully return passages about similar-sounding things. Hybrid search — combining vector similarity with old-fashioned keyword matching — fixes this.
Fixed-size splitting cuts tables in half, orphans list items from their introductions, and separates conditions from their exceptions. The retrieved chunk looks relevant and is missing the qualifying sentence.
Stuffing twenty marginally relevant chunks into the context does not increase the chance of a good answer. It dilutes the signal and invites the model to synthesise across passages that should not be combined.
Retrieve wide, then narrow. Pull 20 to 50 candidate chunks with cheap search, then use a reranking model to pick the best 3 to 5 for the prompt. This two-stage pattern consistently outperforms retrieving five chunks directly, and it costs very little.
Grounding is not automatic. A model handed relevant passages can still drift into recalled knowledge or fill a gap with something plausible. Three habits keep it honest.
A mid-sized company built an assistant over its 400-page employee handbook. Early testing was rough: it answered leave-policy questions confidently and, roughly a quarter of the time, wrongly.
Reading the retrieved chunks made the cause obvious. The handbook had been split every 1,000 characters, so a passage stating an entitlement was frequently separated from the paragraph listing exceptions to it. The model was answering correctly from an incomplete text.
Re-chunking by section heading, keeping each heading attached to its chunk, and adding keyword search for policy codes fixed most of it. They changed nothing about the model or the prompt.
Retrieval must respect permissions. If your index contains documents that not every user may read, the filter has to be applied during retrieval, before anything reaches the prompt. A model asked politely not to reveal a document it has already been given is not an access control system.
RAG suits questions answerable from a handful of passages. It is a poor fit for questions requiring an aggregate view of the whole corpus — how many contracts expire this quarter is a database query, not a retrieval problem.
It is also unnecessary if the relevant material is small. If your entire knowledge base is twenty pages, put all twenty in the prompt and skip the infrastructure entirely.
Chunk documents by structure, embed the chunks, store them somewhere searchable, retrieve widely at query time, rerank down to the best few, and instruct the model to answer only from what it has been given, with citations.
When a RAG system disappoints, inspect the retrieved chunks before touching anything else. The answer quality has a hard ceiling set by retrieval quality.
RAG is best understood as a search problem with a language model attached to the end, not an AI problem with a search box bolted on. Teams who treat it that way spend their effort on chunking, hybrid retrieval and reranking — and they are the ones whose assistants get used.
Start simple, log everything you retrieve, and read those logs. The failures are almost always visible there, in plain text, waiting to be noticed.
Tap a star to share what you thought.
No ratings yet
An architecture where relevant passages are retrieved from your own documents and inserted into the prompt, so a general-purpose language model can answer questions about material it was never trained on. The knowledge lives in a searchable store rather than in the model's weights.
For facts, policies and anything that changes, yes — updates are instant and you get citations. Fine-tuning is the better tool for style, tone, output format and consistent behaviour. Many production systems use both for their respective strengths.
Chunking is splitting documents into retrievable passages. It matters because a chunk is the unit the model actually sees: if a rule and its exceptions land in different chunks, the model will answer confidently from half the picture.
Sign in to join the conversation.
Loading responses…
Have a story, idea, or something valuable to share? Join The Blog Story for free, publish your content, reach more readers, and earn a share of advertising revenue from eligible content.
Create quality content. Grow your audience. Grow your earning potential.
Not usually at first. Postgres with the pgvector extension, or a search engine you already run, handles substantial workloads. Dedicated vector databases earn their place at large scale or when you need specialised indexing features.
Combining semantic vector similarity with traditional keyword search. Vector search is strong on concepts and weak on exact strings like error codes, SKUs and names; keyword search is the reverse. Together they cover both.
A second-stage model that scores retrieved candidates against the question more accurately than the initial search. The standard pattern is to retrieve 20 to 50 candidates cheaply, rerank them, and send only the best few to the language model.
Instruct it to answer only from the supplied passages and to state clearly when they do not contain the answer, require per-claim citations, and show the sources in the interface. Then check the retrieval logs when it still goes wrong.
Apply permission filters during retrieval, so restricted passages never enter the prompt. Instructing the model not to reveal something it has already been shown is not a security control.