Multi-turn RAG — Retrieval-Augmented Generation

Single-question RAG is straightforward: take the question, retrieve, answer. Conversation breaks it immediately. A user asks about holiday carry-over, gets an answer, then asks “what about contractors?” — and that string, searched on its own, retrieves nothing useful.

The fix is to reconstruct what the user meant before retrieving. Everything else in multi-turn RAG follows from that.

Why follow-ups fail

Real follow-ups are short and dependent on what came before:

“What about contractors?”
“Why?”
“Is that still true in 2026?”
“And the second one?”
“Summarise that.”

Each is nearly contentless as a search query. Embedded and searched, “why?” retrieves whatever happens to be nearest to a generic question — effectively random.

Three flavours, worth distinguishing:

Underspecified questions that need the earlier subject supplied. “What about contractors?” means “what is the holiday carry-over policy for contractors?”

Questions about the previous answer. “Why?” or “where did that come from?” may need no retrieval at all — the material is already in the conversation.

Refinements of the earlier retrieval. “Just the 2026 version” is a filter on what was already found.

Handling all three as “embed the user’s text and search” is why conversational RAG often works worse than single-shot.

Query rewriting

The standard approach: before retrieving, rewrite the latest message into a standalone question using the conversation history.

Given the conversation below, rewrite the user's latest message as a
standalone question that can be understood without the conversation.
Keep it faithful — do not add constraints the user did not state.
If it is already standalone, return it unchanged.

“What about contractors?” becomes “What is the holiday carry-over policy for contractors?” — which retrieves properly.

Points that matter in practice:

Use a small, fast model. This is an easy task and it’s on the critical path for latency.

Faithfulness is the failure mode. Rewrites that add unstated constraints send retrieval somewhere the user didn’t ask about, and the resulting answer looks like a non-sequitur. “Do not add constraints” is load-bearing.

Log the rewrite. When a conversational answer is baffling, the rewrite is usually where it went wrong, and you can’t see that without the log.

Retrieve on both, if you can afford it — the rewritten question and the raw message — and merge. Costs a second search and covers rewrite failures.

Keep the original for generation. Retrieve with the rewrite; answer the question the user actually asked. The rewrite is a search artefact, not the user’s words.

Deciding whether to retrieve at all

Not every turn needs a search, and searching anyway wastes latency and can pollute the context.

“Why?”, “explain that differently”, “shorter please”, “what did you mean by X” are usually answerable from the conversation. New retrieval may drag in fresh chunks that shift the subject.

A cheap classification step — does this turn need new information, or is it about what’s already here? — is worth adding. Route accordingly. A wrong decision toward retrieving is mild; a wrong decision toward not retrieving means a genuinely new question gets answered from stale context, which is worse. Bias toward retrieving when unsure.

What to keep in the prompt

The prompt now contains instructions, retrieved chunks, conversation history and the current question. History competes with sources for room, and it accumulates.

Keep the recent turns verbatim — the last two or three exchanges.

Summarise older history rather than truncating it, so earlier context survives in compressed form.

Drop old retrieved chunks. This one is important. Chunks retrieved for turn one shouldn’t sit in the context at turn five; they’re stale, they crowd out current material, and they let the model answer a new question from old sources. Keep the answers, not the chunks.

Bound the total. Decide how much room history gets, and enforce it, rather than letting it grow until something breaks.

Citations across turns

Two things go wrong.

Numbering drift. Source [1] in turn one is a different document from source [1] in turn three. A user scrolling back finds the numbers no longer mean anything.

Either renumber per turn and make that visible, or maintain stable identifiers for the conversation. Either is fine; silently reusing numbers for different documents is not.

References to earlier sources. “As mentioned earlier, [2] says…” when [2] is no longer in context. The model is recalling its own previous output, not the source. If the earlier chunk isn’t present, that claim is unverified — and this is a real path by which grounded conversations become ungrounded over several turns.

Worth instructing against: only cite sources present in the current context.

Drift, and how it happens

The characteristic multi-turn failure: early turns are well grounded, later ones aren’t. Each turn is plausible given the last, and by turn six the conversation has walked away from the documents entirely.

Contributors, all fixable:

Stale chunks in context giving the model old material to draw on.

Rewrites compounding — each one slightly reinterpreting, until the retrieved subject has shifted.

Answering from prior answers rather than from sources, which is a form of the model citing itself.

Mitigations: retrieve fresh for every substantive turn, drop old chunks, apply the same grounding instructions on every turn rather than only the first, and validate attribution per turn.

That last point is the one most often missed. Grounding rules stated once at the start of a conversation get followed less as the conversation lengthens. They belong in every turn’s prompt.