What happens after retrieval
Almost everything written about RAG is about finding the right documents. But retrieval is the first half. Once you have a handful of relevant chunks, a second pipeline begins — assembling them into a prompt, getting the model to actually use them, attaching citations, and deciding what to do when the retrieved material doesn’t answer the question. That second half is where most bad answers come from.
This post is the map of it. Each stage gets its own article elsewhere on this site.
Why the split matters
A RAG system can retrieve perfectly and still answer badly. The chunks are relevant, they contain the answer, and the response is vague, or contradicts them, or cites the wrong one, or confidently answers a question the documents don’t address.
When that happens, teams usually go back and tune retrieval — different embeddings, different chunk size, a reranker. Sometimes that’s right. Often the retrieved set was fine and the problem was downstream, and no amount of retrieval work will fix it.
So the first diagnostic question is always: was the answer present in what we retrieved? If yes, the bug is in the generation half. That one question saves a lot of misdirected effort.
The stages
Selection. You retrieved some number of candidates; not all of them go in the prompt. How many you include changes the answer, and more is not reliably better — extra weakly relevant chunks dilute attention and give the model more opportunities to draw on the wrong thing.
Assembly. The chunks, the question, the instructions and any conversation history get arranged into one prompt. Order matters, formatting matters, and what separates one chunk from the next matters more than people expect, because the model has to be able to tell where one source ends and another begins.
Instruction. You tell the model what to do with the material: answer only from it, cite it, say so if it’s insufficient. The wording of this has a large effect and is one of the cheapest things to improve.
Generation. The model writes. Decoding settings apply here, and for grounded answers the useful settings are different from creative writing.
Attribution. Mapping claims in the answer back to the chunks they came from. Asking the model to cite is easy; getting citations that actually point at the supporting text is a separate problem.
Validation. Checking the answer before the user sees it — that the citations resolve, that nothing was asserted beyond the sources, that no retrieved content is being shown to someone not permitted to see it.
Presentation. What the user actually gets: the answer, its sources, and — importantly — an honest signal when the system couldn’t find enough to answer.
The failure modes, briefly
Each has its own post; this is the shape of them.
The model ignores the context. It answers from its training instead of your documents. Common, usually fixable through instruction and assembly.
The model contradicts the context. Worse, because it looks authoritative. Often caused by the retrieved chunks disagreeing with each other and the model picking one silently.
The model answers anyway. Nothing retrieved supports an answer, and it produces one regardless. Abstention is a behaviour you have to build; models don’t default to it.
Citations that don’t hold up. The answer says a claim came from source 3 and source 3 doesn’t say it. Users trust cited answers more, which makes wrong citations more damaging than no citations.
Instructions from a document. Retrieved content is untrusted input. If a document contains text telling the model to disregard its instructions, and you’ve pasted that document into your prompt, you have handed a stranger a line into your system prompt.
Content shown to the wrong person. Retrieval found a document the asker isn’t allowed to read, and the answer summarised it. Permission checks have to survive into the generation half.
Where the leverage is
Rough order, for a system that retrieves reasonably well already:
- Grounding instructions. Minutes of work, immediate effect on whether the model uses your material.
- Abstention. Deciding and implementing what happens when retrieval comes back thin. Prevents the most damaging class of wrong answer.
- Assembly. Clear chunk boundaries, sensible ordering, sources labelled so citations can refer to them.
- Attribution checking. Mechanical verification that cited sources contain what’s attributed to them.
- Injection and permission handling. Not quality improvements — correctness and safety ones. Do them before you have real users, not after.
Notice that none of these is a retrieval change, and all of them are cheaper than swapping your embedding model.
The short version
RAG is two systems. The first finds material; the second turns material into an answer. The second one gets much less attention and is responsible for most of what users experience as unreliability.
If your retrieval is bringing back the right documents and the answers are still poor, everything you need is in the second half — and it’s the more tractable half.