What to log about a RAG answer

A user reports that an answer was wrong. Without a record of what happened, you cannot tell whether retrieval missed the document, retrieval found it and the model ignored it, the document itself was outdated, or a rewrite sent the search somewhere else entirely. Those have completely different fixes, and you can’t distinguish them from the answer alone.

So the trace is the debugging tool. It is also a copy of your corpus in a system with different access rules, which makes it a security decision as well as an engineering one.

The trace

Per request, enough to reconstruct the whole path:

The user’s question, verbatim.

The rewritten query, if you rewrite. First place to look when a conversational answer is baffling.

Every retrieved candidate — identifier, score, and rank — including the ones you didn’t use. The discarded candidates are frequently the interesting part: if the right chunk was retrieved at rank nine and you include five, that’s a ranking problem, and you can only see it if rank nine was logged.

Which chunks went into the prompt, in the order they appeared.

The assembled prompt, or enough to reconstruct it exactly. This is what settles “did the context even get included” — a question that resolves more “the model ignored my context” reports than anything else.

The model and its version. Answers change when models change, and without this you’ll spend a day on a regression that was an upgrade.

Decoding settings.

The raw output, before any post-processing.

Validation results — which checks ran, what passed, what failed.

The final response as the user saw it, if it differs from the raw output.

Timings per stage. Retrieval, rerank, generation, validation. Tells you where latency lives.

Aggregates worth watching

Individual traces debug one report. Aggregates tell you the system is drifting before anyone reports anything.

Refusal rate. No correct value; a sudden change means something broke — an indexing job, a threshold, a model upgrade.

Validation failure rate, by check. A rise in unmatched quotes is a real signal.

Retrieval score distribution. A shift means your corpus or your queries changed.

Chunks-used versus chunks-retrieved. If you consistently use two of ten, include fewer.

Rank of the chunk that got cited. If citations regularly come from rank seven, your ranking is mis-ordered and your chunk count is compensating.

Latency percentiles per stage, not just the mean. The tail is what users complain about.

Questions with no good retrieval, grouped. This is your best content roadmap — a direct list of what people want that your corpus doesn’t cover.

Reading traces is the point

The habit that pays: read a sample of real traces every week. Not aggregates — actual questions with actual retrieved chunks and actual answers.

You will find things no dashboard shows: chunks that are navigation boilerplate, a document indexed three times, questions phrased in vocabulary your corpus never uses, an entire category of question nobody anticipated. Twenty traces read carefully are worth more than a month of metrics.

Read refusals especially. They’re where the system is telling you what it can’t do.

The privacy cost

Now the part that gets skipped. A full trace store contains:

Your users’ questions, which can be sensitive in themselves — what someone asked about a medical policy, a grievance procedure, a redundancy process.

Retrieved document content, in full. This is the significant one. Your trace store is now a partial copy of your corpus, and if it’s a general logging system, everyone with log access can read documents they may have no rights to.

That’s the same disclosure problem that permission-filtered retrieval exists to prevent, reintroduced through the back door.

Handle it deliberately:

Restrict trace access to the corpus’s access level, or don’t store chunk text. Store identifiers and retrieve content on demand from the permissioned store when investigating.

Set retention and enforce it. Traces are valuable for days and liability for years. Thirty days of full traces plus long-term aggregates covers nearly every real need.

Separate the stores. Full traces in a restricted system, metrics and timings in your ordinary observability stack. Most of what you look at daily is the second kind.

Redact questions where required. Depending on your setting, user questions may be personal data with obligations attached.

Sample rather than storing everything, at volume. Full traces on a percentage of requests plus identifiers-only on the rest gives you debugging ability at a fraction of the exposure.

Keep traces out of third-party services unless that’s a decision someone made knowingly. Sending prompts to an external observability vendor sends your documents there too.

A workable default

Identifiers, scores, ranks, timings, model version, and validation results on every request, in your normal observability system, retained as long as you like — none of that is document content.

Full prompts and chunk text on a sample, in a restricted store, retained for thirty days, access-controlled to match the corpus.

That combination lets you investigate almost anything, catch drift early, and not accidentally build an unpermissioned copy of every document you index.