How many chunks to include

Given a ranked list of retrieved chunks, you have to decide how many go into the prompt. The instinct is that more context means more chance the answer is in there. That’s true for retrieval, and it stops being true for generation: past some point, extra chunks make answers worse rather than better.

Finding roughly where that point is, for your corpus and your questions, is one of the higher-value tuning exercises available.

Why more stops helping

Dilution. Every additional chunk is more material competing for the model’s attention. A prompt with three relevant chunks concentrates attention on them; the same three plus twelve marginal ones don’t.

Position effects. Longer contexts make the middle-of-the-prompt weak spot more pronounced. Adding chunks can push a good chunk from a strong position into a weak one.

More chances to be wrong. A marginally relevant chunk that mentions the right topic but the wrong case is a trap. It reads as on-topic, and it supplies a plausible wrong answer. With three chunks the model is unlikely to reach for it; with fifteen, one of them is probably like this.

Conflict without resolution. Broad retrieval increases the chance of including sources that disagree. That’s good if your system detects and reports conflict, and bad if it silently picks.

Harder attribution. Citations get less precise as candidates multiply, because more sources plausibly support any given claim.

The shape of the curve

Informally, answer quality against number of chunks tends to rise steeply for the first few, flatten, and then decline slowly. The interesting part is where the flat region begins — that’s the number you want.

Where it falls depends on things you can reason about:

Chunk size. Large chunks reach the flat region sooner, because each one carries more.

How focused your questions are. A question with one answer in one place needs few chunks. A question requiring synthesis across sources needs more.

Retrieval precision. If your top result is usually right, few chunks suffice. If the right answer is often ranked fifth, you need enough depth to include it — but the real fix there is ranking, not volume.

Whether you rerank. A good reranker means your top few are genuinely the best few, so a small number goes further.

There isn’t a universal right answer, but the honest generalisation is that most systems use more than they need, and that a small number of well-ranked chunks beats a large number of loosely ranked ones.

Finding your number

You don’t need elaborate machinery:

Take twenty or thirty real questions with known correct answers. Real ones, from actual users if you have them — invented questions are systematically easier than real ones and will mislead you.

Run each at several values — say 3, 5, 10, 20 chunks — holding everything else fixed.

Read the answers. Not a score, the actual text. You’re looking for the pattern of failures, which is more informative than a number: are answers wrong because the material was missing, or because it was present and the model used the wrong part? The first says increase; the second says decrease.

Pick the smallest value where quality stops improving. Not the value with the best score — the smallest one that’s indistinguishable from the best. Smaller is better for latency, and it makes attribution cleaner.

Adaptive selection

A fixed number is a reasonable default, but the number of useful chunks varies by question, and you usually have a signal for it: the retrieval scores.

Two approaches, both better than a constant:

Score threshold. Include chunks above some relevance score, up to a maximum. A question with one strong match includes one chunk; a question with several close matches includes several. Requires that your scores be comparable across queries, which is worth verifying rather than assuming.

Score gap. Include chunks until there’s a large drop in score. Handles the common case where the top three are close and the fourth is much worse.

Both give you fewer chunks on easy questions and more on hard ones, which is the right behaviour and also faster on average.

What a thin result should mean

The important consequence of adaptive selection: sometimes it returns almost nothing, and that is information.

If your best retrieved chunk has a weak score and nothing else is close, the honest response is usually “I don’t have material on this” rather than an answer built from the least-bad thing available. A system that always fills its context to capacity has thrown that signal away.

So the number-of-chunks decision connects directly to abstention. The two questions “how much do I include?” and “when do I decline to answer?” are the same question asked twice.

Reasonable defaults

If you need somewhere to start rather than an experiment: three to five chunks of a few hundred words each, ranked, with the strongest nearest the question, and a score threshold that can return fewer.

Then run the twenty-question exercise, because the whole point is that the right value depends on your corpus — and the exercise takes an afternoon and tells you more than any general recommendation including this one.