When one retrieval pass isn't enough

A single retrieval pass answers questions whose answer sits in one place. Plenty of real questions don’t: comparisons across documents, multi-step lookups where the first answer determines the second search, and questions whose phrasing shares no vocabulary with the passage that answers them.

For those, the system needs to search more than once — and the interesting part is not the searching, it’s knowing when to stop.

Questions that need more than one pass

Multi-hop. “What is the notice period for the role Priya is in?” You have to find Priya’s role, then find the notice period for that role. No single passage contains both, and the second query can’t be written until the first is answered.

Comparison. “How does the 2024 policy differ from the 2026 one?” Needs both, and a single search usually surfaces one strongly and the other weakly.

Aggregation. “Which teams have deployment runbooks?” The answer is distributed across many documents and no chunk states it.

Vocabulary mismatch. The user asks about “time off”; the document says “annual leave entitlement”. Sometimes a rephrased second search finds what the first missed.

Thin first results. Retrieval came back weak. Rather than answering badly or refusing, try a different query.

The loop

The shape is simple:

1. Retrieve for the current question.
2. Ask the model: is this sufficient to answer? If yes, answer.
3. If not, ask what specifically is missing, and generate a query for it.
4. Retrieve again, add to the accumulated context.
5. Repeat, up to a limit.

Everything difficult is in steps 2 and 3, plus the limit.

Loop control

This is where iterative retrieval goes wrong, and it goes wrong expensively — each iteration is a retrieval plus at least one model call, on the critical path.

Hard iteration cap. Two or three. Not five. If three passes haven’t found it, a fourth usually won’t either, and you’ve now spent several seconds and several calls to produce a refusal you could have given at the start.

Stop when a query repeats. A loop that keeps generating the same query has converged on nothing. Normalise and compare against previous queries; if it repeats, stop.

Stop when nothing new arrives. If a pass returns only chunks you already have, further passes won’t help.

Total budget, not just iteration count. Cap the wall-clock time and the number of model calls. An iteration that happens to be slow shouldn’t be allowed to blow the whole request.

A refusal is a legitimate outcome. The loop’s job is to find the answer or establish that it isn’t there. Having searched three ways and found nothing is better evidence for “I don’t know” than a single pass — and that’s a real benefit, not a failure.

Asking what’s missing

The quality of step 3 determines whether the loop is useful.

Vague: “the sources are insufficient.” Nothing to act on.

Useful: “I found Priya’s role — Senior Engineer — but nothing about notice periods for that role.” Now the next query writes itself.

So instruct for specificity:

If the sources are insufficient, state exactly what additional
information is needed, then write a single search query that would
find it. Do not repeat a query you have already tried.

Giving the model its previous queries matters. Without them it regenerates near-duplicates and the loop spins.

Accumulating context

Each pass adds chunks, and the context grows through exactly the region where more isn’t better.

Deduplicate. Passes overlap. The same chunk arriving twice wastes room and skews attribution.

Keep a running relevance judgement. Not everything retrieved in pass one is still relevant once pass two clarified the question. Dropping chunks that no longer matter is worth the effort on longer loops.

Watch the total. Three passes of five chunks is fifteen chunks — likely more than you want. Consider narrowing to the best few before the final generation.

Preserve the trail for logs. Which pass found what. Essential for debugging, and it doesn’t have to be in the prompt.

What it costs

Be honest about the trade, because this is a significant one:

Latency multiplies. Each pass is a retrieval plus a sufficiency judgement, sequentially. A two-pass question takes roughly twice as long as a one-pass question, and users notice seconds.

Cost multiplies, similarly.

Variance increases. Some questions finish in one pass, some in three. Your latency distribution develops a long tail, which is harder to operate around than a slower but predictable system.

More places to fail. Each pass can go wrong. A bad generated query can pull the context off-subject.

Which is why iterative retrieval should be conditional rather than default.

Making it conditional

Run one pass. If the model judges the material sufficient — which for most questions it will — answer and stop. Only iterate when the first pass genuinely came back short.

That gives you single-pass latency for the common case and multi-pass capability for the hard case, which is the right shape. It also means the sufficiency judgement in step 2 is doing double duty: it’s your loop control and your abstention signal.

Whether you need it at all

Before building a loop, check the cheaper options:

Better first-pass retrieval. Some multi-pass need is really a ranking problem. If the answer was in your index at rank eight, fix the ranking.

Query expansion in a single pass. Generate several query variants, search them all in parallel, merge. Handles vocabulary mismatch without a sequential loop, and parallel searches don’t multiply latency the way sequential ones do.

Larger chunks. Multi-hop questions sometimes become single-hop when the two facts land in the same chunk.

Genuinely multi-hop questions — where the second query depends on the first answer — do need iteration. Everything else probably doesn’t, and the cheaper fixes are worth exhausting first.