Teaching a model to say I don't know
Language models are built to produce a plausible continuation, and “I don’t know” is rarely the most plausible continuation of a question. So a RAG system that retrieves nothing useful will still answer, using whatever the retrieved material vaguely suggested plus whatever the model already believed.
Abstention has to be built. It’s the single highest-value reliability feature in the generation half, and it’s also the easiest to over-tune into uselessness.
Why it matters more here than elsewhere
A general chatbot that guesses is annoying. A RAG system that guesses is worse, for a specific reason: users trust it more. The whole promise is that answers come from your documents. An answer that didn’t, presented identically to ones that did, is a confident falsehood wearing the uniform of a sourced fact.
And the failure is invisible from the outside. Nothing about the response indicates the sources were thin.
Three places the decision can live
Before generation, on retrieval signals. If the best retrieved chunk scores poorly, or nothing clears a threshold, stop and return a “nothing found” response without calling the model.
Cheap, fast, and completely reliable at what it does. Its weakness is that retrieval scores are an imperfect proxy for whether the answer is present — a weak score can still accompany the right passage, and a strong one can accompany a topically similar but wrong passage.
During generation, by instruction. The model is told it may decline and does so when the material doesn’t support an answer.
More semantically aware than a score threshold — the model can tell that a chunk is about the right subject but the wrong year. Less reliable, because it depends on the model actually following the instruction, which it does most of the time rather than always.
After generation, by validation. Generate the answer, then check whether its claims are supported by the sources. If not, suppress it.
The most accurate and the most expensive, since you’ve already paid for a generation you’re throwing away.
Use all three. They fail differently, which is exactly why layering them works: the threshold catches the empty-retrieval case for free, the instruction catches semantic mismatch, and validation catches the instruction being ignored.
Getting the threshold right
The pre-generation threshold is where over-tuning happens. Set it too high and the system refuses questions it could have answered, which users experience as the system being useless — and unlike a wrong answer, they’ll tell you about it immediately.
Calibrate on real questions. Take questions you know are answerable and questions you know aren’t, run both, and look at the score distributions. If they overlap heavily, the threshold can’t separate them and you should lean on the other two layers instead.
Verify scores are comparable across queries. Some similarity measures produce scores whose absolute values shift with query length or phrasing. A fixed threshold on a non-comparable score is close to arbitrary.
Prefer the score gap. “Is the top result much better than the rest?” often generalises better than “is the top result above 0.8?”.
Accept asymmetry deliberately. Which error is worse depends on what the system does. For an internal tool where users can ask a colleague, refusing is cheap. For anything giving consequential guidance, refusing is much cheaper than being wrong. Decide this on purpose rather than inheriting whatever the default threshold was.
Write a good refusal
A refusal is a real response and deserves as much thought as an answer. A bare “I don’t know” is a dead end.
Include, where you can:
What was searched. “I looked through the policy documents and didn’t find anything on this.” Tells the user the scope, which is often the actual missing information — they may be asking a system that simply doesn’t have that corpus.
What was close. “The closest material covers the previous version of this policy.” Frequently the most useful part, and sometimes it’s the answer.
What would be needed. “This would be in the finance handbook, which isn’t indexed here.”
Where to go next. A human, a different system, a place to look.
Compare “I don’t have information on that” with “I searched the engineering documentation and found nothing about the deployment schedule — the closest match was the release process from 2024. This may be in the operations runbook, which isn’t included here.” Same refusal; the second is useful.
Partial answers
Many questions are partly answerable, and forcing a binary choice loses information.
The right behaviour is to answer the supported part and be explicit about the rest: “The sources confirm the limit is ten days. They don’t say whether unused days expire at year end.” That’s more useful than either a full refusal or a complete answer with an invented second half.
This needs to be instructed for, since the natural failure is to answer both parts with equal confidence:
Answer the parts of the question the sources support. For any part
they do not address, say explicitly that the sources do not cover it.Watch the rate
One number worth logging: what fraction of questions get a refusal.
There’s no correct value, but the trend and the extremes are informative. A rate near zero means abstention isn’t functioning — with real users, some questions are always unanswerable. A rate that’s very high means your threshold is too aggressive, or your corpus doesn’t match what people ask, and the distinction between those two matters.
A sudden change is worth investigating: it usually means an indexing job failed, a model was upgraded, or the kind of question being asked shifted.
And the refusals themselves are the most valuable log you have. Read them regularly. They’re a direct list of what people want that your documents don’t cover, which is the best content roadmap available for whatever your corpus is supposed to be.