Grounding instructions that work

The instruction block at the top of a RAG prompt is usually three or four sentences written once and never revisited, and it has more effect on answer quality than most retrieval tuning. It’s where you establish that the sources are binding, that citations are required, and that “I don’t know” is an acceptable output.

Here’s what belongs in it and why each part is there.

The five things it needs to say

Sources are the only permitted basis. Not “use the context to help” — “answer using only the sources below”. The difference between a suggestion and a constraint.

Not knowing is allowed. Models default to producing an answer. If declining isn’t explicitly permitted, it effectively isn’t available, and thin retrieval turns into invention.

Citations are required, in a stated format. Specify the format exactly, or you’ll get four different ones.

What to do when sources conflict. Otherwise the model picks one silently and you never find out there was a disagreement.

What the answer should look like. Length and shape, briefly. Unspecified, you get an essay when you wanted a sentence.

Five sentences, roughly. Each one prevents a distinct failure.

A working version

Answer the question using only the sources provided below.

If the sources do not contain enough information to answer, say
"I don't have information on that" and stop. Do not fill gaps with
general knowledge.

Cite every factual claim with the source number in square brackets,
like [1]. If a claim draws on two sources, cite both.

If the sources disagree, say so explicitly, give both positions with
their citations, and prefer the more recent source where dates are
available.

Answer in at most three short paragraphs. Do not repeat the question
back.

Nothing clever. But it addresses each of the five, and it’s specific enough to be followed rather than interpreted.

Techniques worth adding

Quote before answering. For anything where correctness matters:

Before answering, quote the sentence or sentences from the sources
that support your answer. Then give the answer.

The strongest single grounding technique available. A model that must produce a supporting quote can’t answer from memory, because the quote has to come from somewhere. It also makes review dramatically easier — you can check the quote against the source mechanically.

Cost: a longer response, and you may not want the quotes shown to users. Generate them and strip them before display, or keep them for logs.

Name the scope. Where your sources are specifically your organisation’s, saying so prevents the model from substituting the general public answer:

The sources describe this organisation's own policies, which may
differ from common practice. Follow the sources.

Ask for the gap. Instead of only permitting refusal, ask what’s missing:

If the sources are insufficient, state what specific information
would be needed to answer.

Turns a dead end into something actionable, for the user and for you — repeated gap reports are a reading list of what your corpus lacks.

Separate confidence from content. Where downstream systems need it, ask for an explicit signal:

End with a line: Confidence: high | medium | low, based only on how
directly the sources address the question.

Treat this as a rough signal, not a probability. It’s useful for routing — low-confidence answers get a warning or a human — and not for anything requiring calibration.

Things that don’t help

Emphasis. Capitals, “IMPORTANT”, exclamation marks, “you MUST”. Clear phrasing does the work; volume doesn’t add to it.

Threats and inducements. No.

Piling on redundant instructions. Three sentences saying the same thing dilute each other. One clear sentence per requirement.

Very long instruction blocks. Past a point, instructions compete with your actual sources for attention, and later instructions get followed less reliably than earlier ones. If your block is approaching a page, some of it isn’t earning its place.

Role-play framing. “You are a world-class expert researcher” doesn’t improve grounding. Instructions about the task do.

Order within the block

Put the constraints first — sources-only, refusal permitted — then citation format, then conflict handling, then output shape. Earlier instructions are followed more reliably, so the correctness rules should precede the cosmetic ones.

Keep the instructions above the sources, not below them. Below a long context block, they’re in the weak middle region of the prompt, and they’re also being read after the material they were supposed to frame.

Test changes properly

The tempting failure mode here is editing the instruction block, trying one question, seeing a better answer, and shipping.

Keep a fixed set of twenty or thirty real questions, and run the whole set before and after any change. Include questions your sources can’t answer — those are the ones that reveal whether your refusal instruction is working, and they’re exactly the ones nobody thinks to test.

Instruction changes have broad effects. A tightening that improves grounded answers can also make the model refuse questions it should have answered, and you’ll only see that if unanswerable and answerable questions are both in your set.

The minimum

If you take one thing: change “use the following context to help answer” to “answer using only the sources below, and say you don’t know if they don’t contain the answer.”

Two clauses. It’s the highest-value edit in the generation half of most RAG systems, and it takes longer to find the file than to make the change.