Assembling the prompt
Once you have your chunks, they have to be arranged into a single prompt alongside the question, your instructions and any conversation history. This assembly step is usually written once, early, and never revisited — which is a shame, because it’s one of the cheapest places to improve answer quality.
Three things matter: what order the pieces go in, how clearly the sources are separated, and whether the model can tell which source is which.
Order of the pieces
A reliable arrangement:
- The task instructions — what to do, how to cite, when to decline.
- The retrieved context — the chunks, labelled.
- The conversation history, if any.
- The user’s current question, last.
The reasoning: instructions first because they frame everything after them, and the question last because it’s what the model should be holding when it starts writing. Putting the question before a large block of context means the model reads the question, then reads thousands of tokens, then answers — and the question has become distant.
Some people put the question both before and after the context. This sounds redundant and often helps, particularly with long contexts.
Separating the sources
The model needs to be able to tell where one chunk stops and the next starts. If chunks are concatenated with a blank line, they read as one continuous document, and the model can happily merge a sentence from one source with a sentence from another into a claim that neither made.
Use an unambiguous delimiter and a label:
[1] Source: employee-handbook.pdf, section 4.2
Employees may carry over up to five unused days...
[2] Source: policy-update-2026.md
As of January, the carry-over limit is ten days...Three things this gives you:
A referent for citations. The model can say “according to [2]” and mean something checkable.
Visible provenance. A filename and a section tell the model — and later the user — where a claim comes from.
Detectable conflict. In the example above, the two sources disagree. Clearly separated and labelled, that’s something the model can notice and report. Merged into one block, it silently picks one.
Whatever format you choose, keep it consistent, and make sure the delimiter can’t appear inside your documents.
Position within the context
Models don’t attend evenly across a long prompt. Material at the beginning and end of the context tends to be used more reliably than material in the middle — an effect documented in the research literature and easy to reproduce informally. It gets more pronounced as the context gets longer.
Practical consequences:
Don’t bury the most relevant chunk in the middle. If your retrieval produces a ranking, the ordering of chunks in the prompt is a decision, not an accident.
Consider putting the strongest chunk last, closest to the question. Or first. Both positions are better than the middle.
Shorter contexts sidestep the problem. Ten mediocre chunks are often worse than three good ones, and not only because of cost.
Test rather than assume. The effect varies by model and changes between versions. If your prompt assembly quietly depends on a particular model’s attention pattern, that’s worth knowing when you upgrade.
Metadata worth including
Alongside the chunk text, small pieces of context change how usable the material is:
Source identity — the document name. Needed for citation.
Location within the document — page, section, heading. Lets you link the user to the right place, not just the right file.
Date — the single most valuable extra field. Without it, a model reading an old policy and a new one has no way to prefer the new one. With it, you can instruct it to.
Document type or authority level, if your corpus has a hierarchy — an official policy versus a discussion thread. Lets the model weight sources sensibly rather than treating everything as equally authoritative.
Keep this compact. Metadata competes with content for room in the prompt, and a verbose header on every chunk adds up fast.
What to do when it doesn’t fit
Sooner or later your selected chunks exceed the room you’ve allowed. Options, roughly in order of preference:
Include fewer chunks. Usually the right answer. Dropping the weakest candidates rarely costs you the answer, because if the answer were in the weakest candidate your ranking has bigger problems.
Truncate individual chunks, keeping their beginnings. Acceptable, though it can cut a sentence mid-claim.
Summarise chunks before including them. Adds a model call and a place for errors to be introduced, but useful when you genuinely need coverage across many documents.
Two-pass retrieval — a first pass narrows, a second goes deeper on what survived.
What to avoid is silently dropping chunks with no policy, which produces answers that vary unpredictably depending on how long the retrieved documents happened to be.
A template to start from
Answer the question using only the sources below.
Cite the sources you use as [1], [2], etc.
If the sources do not contain the answer, say so and do not guess.
Prefer more recent sources when they conflict, and note the conflict.
Sources:
[1] Source: <name>, <location>, dated <date>
<chunk text>
[2] Source: <name>, <location>, dated <date>
<chunk text>
Question: <the user's question>Unremarkable, and it handles ordering, separation, labelling, citation, abstention and conflict. Most of what follows on this site is about doing each of those parts better — but this is a real starting point, and it beats concatenation.