Arithmetic the sources didn't do

A user asks how much was spent across three regions. Retrieval returns three regional figures. The model adds them up and reports a total. That total appears in none of the sources, and nothing in the answer distinguishes it from the figures that do.

Derived numbers are the quietest failure in the generation half. Every input is correct, every citation points somewhere real, and the headline figure is still something no document ever said.

A derived claim can’t be cited

Sums, differences, percentage changes, averages, unit conversions, rankings and durations between dates are all new claims. The citations attached to them refer to the inputs, not the result.

That breaks validation in both directions. A checker looking for the total in the source text flags it as unsupported, so you learn to ignore the flag. Or the checker is lenient enough to see all the component numbers present in the context and wave the whole answer through — which is worse, because now the one claim that was never in a document is the one claim nobody looked at.

Where it actually goes wrong

Unit mismatches. One chunk reports in thousands, another in whole units, and the row header that said so was in a part of the table that didn’t get retrieved. The arithmetic is flawless and the answer is off by a factor of a thousand.

Different periods. One figure covers a quarter, one a year. Both are labelled “spend”.

Different definitions. Headcount including contractors added to headcount excluding them. The words match; the quantities aren’t comparable.

Missing rows. Retrieval returned three of five regions. The model totals what it was given and presents a confidently partial figure. This is the most dangerous case because nothing about the answer is detectably wrong — the arithmetic is right and the citations check out.

Token-by-token computation. Models produce digits as text, not as the output of a calculator. Long multiplication, compound percentages and multi-step conversions fail in ways that look entirely plausible, and they fail intermittently, which is worse than failing consistently.

Three defensible policies

Pick one deliberately rather than discovering which one you have.

Don’t derive at all. The model reports the figures it found and stops. Cheapest, most honest, and frequently what the user needed anyway.

Derive, but show the work. Each input is stated with its citation, then the operation, then the result explicitly labelled as computed rather than quoted.

Compute outside the model. Extract the figures as structured data, do the arithmetic in code, and hand the result back for the model to phrase. See structured output from retrieved context for the extraction half. This is the right answer whenever the number matters, because it moves the calculation somewhere you can unit-test.

Instructing for the middle option

Do not calculate totals, differences, averages or percentages
unless the question requires it.

If you must calculate:
- List each input figure with its citation and its units.
- State the operation you performed.
- Label the result as calculated, not quoted.
- If the inputs use different units, periods or definitions,
  do not combine them; say why.
- If you cannot confirm you have every relevant figure, say
  that the result may be incomplete.

The last two lines carry most of the value. They convert two silent failures into visible ones.

Completeness is the harder half

Unit and period mismatches are visible in the prompt if you label chunks well. Missing rows are not — the model cannot tell the difference between “these are the three regions” and “these are three of the regions.”

What helps:

Make aggregate questions a recognised shape. Words like total, how many, across all, average and combined signal that completeness matters. Those queries deserve a different path, often a broader or second retrieval pass — see more than one retrieval pass.

Prefer a source that already totals. If a document contains the total, retrieve and quote it. A quoted total is a grounded claim; a computed one isn’t.

Have the model enumerate what it found. “I found figures for three regions: A, B and C” lets a reader who knows there are five catch the problem instantly.

Precision, units and currency

Carry the source’s precision through. If a document says “about 40”, the answer should not say “40.0”, and a percentage derived from two rounded inputs should not gain decimal places it never had.

Keep units and currency symbols attached to every figure, in the answer as well as in the prompt. And don’t convert currencies: the exchange rate isn’t in your corpus, so a converted figure is grounded in nothing at all.

Date arithmetic deserves the same suspicion. “Thirty days from the date of notice” requires knowing the date of notice and today’s date, and the model reliably has neither unless you supplied them.

Route derived figures to a different review queue

Answers containing a computed number are a distinct risk class, and they are easy to detect — you know whether your instruction block permitted a calculation, and you can flag answers whose numbers don’t appear verbatim in the retrieved text. Add that flag to whatever you already capture in what to log about a RAG answer, and sample from it.

The honest version of an aggregate answer is often less satisfying and much more useful: here are the figures I found, here is where each came from, and I haven’t added them up because I can’t confirm this is all of them.