Why the model ignores your context

A specific and common failure: retrieval works, the correct passage is sitting in the prompt, and the model answers from its training data instead. Sometimes the answer is even right — just not sourced from your documents, which means next time it won’t be.

This is a grounding failure, and it usually has one of a few identifiable causes.

First, confirm that’s what happened

Before diagnosing, check the answer was actually ungrounded rather than the retrieval being wrong.

Log the exact prompt you sent, and read it. Not the chunks your retriever returned — the final assembled string. Then find the answer’s claim in it.

The three possibilities:

The claim isn’t in the prompt. Retrieval problem, not a grounding problem.

It’s in the prompt and the answer matches it. Working correctly; the answer just also resembles common knowledge.

It’s in the prompt and the answer differs. This is the failure. Continue.

Reading the assembled prompt sounds obvious and is skipped constantly. A surprising share of “the model ignored my context” turns out to be a template bug where the context was never included at all.

Cause: the instruction is too soft

The most common cause. Instructions like “use the following context to help answer” describe the context as optional assistance, and the model treats it that way.

Compare:

Use the following context to help answer the question.

Answer using only the sources below. Do not use any other knowledge.
If the sources do not contain the answer, say that you don't know.

The second is a constraint. The first is a suggestion. This single change fixes a meaningful fraction of grounding problems and takes a minute.

Push it further where correctness matters: ask for the supporting quote before the answer. A model that has to produce the sentence it’s relying on is much less able to answer from memory, because there’s nowhere to get the quote from.

Cause: the question is common knowledge

Models ground more readily on obscure material than on familiar material. Ask about your company’s internal leave policy and the model has nothing else to draw on. Ask about a widely documented public topic and its training has a strong, fluent answer ready.

This is why grounding failures cluster on questions that sound general even when your documents contain a specific, different answer. Your organisation’s definition of a term, your product’s particular behaviour, your local variation on a standard practice.

Handling: make the specificity explicit. “According to the sources provided, which describe this organisation’s own policy, …” tells the model that the general answer is not the one being asked for.

Cause: the context is buried

A relevant chunk in the middle of a long prompt gets used less reliably than the same chunk near the start or end. If you’re including many chunks and the right one is somewhere in the middle, “ignored context” may be a position problem.

Handling: fewer chunks, and the strongest one adjacent to the question.

Cause: the context is hard to read as authoritative

Retrieved text arrives in whatever shape the source document was in. If a chunk is a fragment of a table, a navigation menu, boilerplate, or a mid-sentence cut, the model may reasonably decline to treat it as a statement of fact.

Handling: this is really an ingestion problem surfacing at generation time, and the fix is upstream — cleaner extraction, chunks that end at sensible boundaries. But you can check for it quickly by reading a few of your actual chunks. If you wouldn’t be confident answering from them, neither is the model.

Cause: the format hides the boundaries

Chunks concatenated with no delimiters read as one document. The model has no way to know which claim came from where, so it can’t cite, and its handling of the material becomes vaguer.

Handling: labelled, delimited sources, as in the assembly template.

Cause: conflict with what it believes

Occasionally the retrieved material contradicts something strongly represented in the model’s training — a policy that changed, a number that was revised, a name that’s different now. Some models will quietly prefer the familiar version.

Handling: this is the case for explicit instruction that the sources supersede prior knowledge, plus dates on your chunks so recency is visible. It’s also the case for verification: if your validation step checks that assertions appear in the sources, a preference for training data becomes detectable rather than invisible.

The order to try things

1. Read the assembled prompt. Rules out the template bug and tells you which failure you have.
2. Tighten the instruction to “only”. Cheapest real fix.
3. Ask for the supporting quote. Strongest single lever for grounding.
4. Reduce chunk count, move the best one adjacent to the question.
5. Add clear source delimiters and labels.
6. Read a sample of your actual chunks for the quality problem.
7. Add dates and a recency instruction if the corpus has superseded material.

Notice none of these is a retrieval change. When the right passage is already in the prompt, retrieval is not the thing to fix — and the whole list above is a day’s work at most.