Decoding settings for grounded answers

Sampling settings — temperature, top-p, penalties, maximum length — control how the model picks each next token. For creative writing, higher randomness produces more interesting text. For grounded answers, the goal is the opposite: reproduce what the sources say, with as little invention as possible.

The defaults in most libraries are tuned for general chat, and they’re not what you want.

Temperature

Temperature scales how much the model favours its highest-probability next token. Low values make it near-deterministic; high values flatten the distribution and let less likely tokens through.

For RAG, low. Somewhere between 0 and 0.3.

The reasoning is direct: when the answer is supposed to come from a passage in the prompt, the highest-probability continuation is usually the one closest to that passage. Randomness is licence to deviate from the source, which is exactly the behaviour you’re trying to prevent.

Temperature 0 for extraction, figures, definitions, structured output — anything where there’s one right answer and you want it reproducibly.

Around 0.2 for explanatory answers, where slightly more natural phrasing is worth a small amount of variability.

Above 0.5 essentially never, for grounded answers. If you find yourself raising temperature to make answers better, the problem is upstream — a weak prompt, poor chunks, insufficient instruction — and randomness is masking it.

Note that temperature 0 is not truly deterministic in practice. Batching, hardware and floating-point non-associativity mean identical requests can produce different outputs. It’s near-deterministic, which is different from reproducible — worth knowing before you write a test asserting exact output.

Top-p and top-k

Both restrict which tokens are candidates before sampling: top-p keeps the smallest set whose cumulative probability exceeds a threshold, top-k keeps a fixed number.

Adjust one, not both, and generally leave them alone if temperature is already low — at low temperature the distribution is concentrated anyway and these have little left to do.

If you prefer to control truncation rather than temperature, a top-p around 0.1 with temperature 1 gives similar conservatism. Setting both aggressively is redundant and makes the interaction hard to reason about.

Frequency and presence penalties

These discourage repeating tokens that have already appeared. They exist because chat models can fall into loops.

For RAG, keep them at zero or very low.

A grounded answer should repeat terms — the terms in the source documents. Penalising repetition pushes the model toward synonyms, and synonym substitution in a grounded answer is drift. “Annual leave” becoming “vacation time” is harmless; “may be eligible” becoming “qualifies” is a changed claim.

If you have a genuine repetition problem, it’s usually a prompt or chunk problem — duplicate chunks in context, or an instruction that invites restating the question.

Maximum length

Set it, deliberately, and higher than you expect to need.

Too low is the more damaging error. An answer truncated mid-sentence is bad; an answer truncated after the claims and before the citations is worse, because you’ve lost the attribution while keeping the assertions. Validation will then flag uncited claims that were never uncited.

Guidance:

Account for the quote-then-answer pattern. Asking for supporting quotes before answers roughly doubles output length. A limit tuned before you added quotes will now truncate.

Control length by instruction, not by the limit. “Answer in at most three short paragraphs” produces a complete short answer. A tight token limit produces an incomplete long one.

Treat hitting the limit as an error worth logging. If responses are regularly truncated, something is wrong — usually an instruction that isn’t constraining length.

Stop sequences

Useful if your prompt has structure the model might continue past — if you’re asking for a specific format, a stop sequence at the boundary prevents it from writing a second example.

Make sure the sequence can’t appear in legitimate output. A stop sequence that occurs inside a quoted document truncates the answer for no reason, and it’ll be intermittent and confusing.

Reasonable starting points

For a factual question-answering system: temperature 0.1, top-p and penalties at defaults, maximum length generously above your expected answer, plus quotes.

For extraction or structured output: temperature 0, and validate the output against your schema.

For an explanatory or tutorial-style system: temperature 0.2 to 0.3, and lean on instructions for tone.

Then leave them alone. These settings are a small lever, and the temptation is to fiddle with them because they’re easy to change — a number in a config file, versus rewriting a prompt or improving chunking.

What they can’t fix

Worth being clear, since decoding settings attract more attention than they deserve:

They can’t make the model use context it’s ignoring. That’s instruction and assembly.

They can’t prevent hallucination. Temperature 0 makes the model deterministically produce its most likely output, which can be confidently wrong. Low temperature reduces variance, not error.

They can’t fix bad retrieval. No sampling configuration extracts an answer from chunks that don’t contain it.

They can’t substitute for validation. A low-temperature answer still needs its citations checked.

Low temperature is the correct setting and it’s a small effect. The large effects in the generation half are the instruction block, abstention, assembly and validation. If decoding settings are where you’re spending your tuning time, there’s almost certainly more available elsewhere.