Summarising across many documents

“What are the common themes in last quarter’s support tickets?” isn’t a retrieval question. There’s no passage containing the answer; the answer has to be constructed from many documents, more than fit in one prompt.

The standard approach is staged: summarise in batches, then summarise the summaries. It works, and each stage loses a little fidelity, so the design question is where to spend effort keeping accuracy.

Recognising the question type

Distinct from ordinary RAG questions, and worth detecting because the handling is different:

Aggregation. “How many”, “which teams”, “what proportion”.

Theme extraction. “Common complaints”, “recurring issues”.

Comprehensive listing. “All the policies that mention X”.

Comparison across a set. “How do our regional policies differ?”

The tell is that no single document could answer it. Ordinary retrieval handles these badly — it returns the five most relevant documents and the model produces a confident summary of five out of two hundred, with no indication that’s what happened. That’s the most misleading failure in this whole area, because the answer looks complete.

The staged pattern

Stage one: select the set. Not “the top five” — everything matching. This is a filter, not a ranked search: all tickets in the date range, all documents with a tag. Metadata filtering does this, not vector similarity.

Stage two: process in batches. Split into groups that fit comfortably in context. For each batch, extract what the question needs — not a general summary, but the specific thing being asked for.

Stage three: combine. Feed the batch results into a final call that produces the answer.

Stage four, if needed: repeat stage three. With very large sets, batch results themselves won’t fit, so combining becomes hierarchical.

Batches can run in parallel, which matters — this pattern is slow, and the batch stage is the parallelisable part.

Where accuracy leaks

Each stage is a lossy compression, and knowing where the loss happens tells you where to intervene.

Instruct batch extraction narrowly. A general “summarise these tickets” drops exactly the details the question needed. “List every distinct complaint mentioned, with the ticket ID” preserves them. The batch prompt should be about the question, not about summarising.

Keep identifiers through every stage. If a batch result says “three tickets mention slow exports” without saying which, the final answer can’t cite anything and you can’t verify it. Identifiers are what make the output checkable at all.

Preserve counts, don’t recompute them. Models are unreliable at arithmetic over text. Have each batch report its counts, then sum them in code. Any number in the final answer should be computed, not generated.

Watch for double counting. The same issue described in two tickets is two mentions; the same ticket in two batches is a bug. Deduplicate the set before batching.

Beware smoothing at the combine stage. Combining tends to produce plausible generalities and drop outliers. If the question is about the unusual case, say so explicitly — “include items mentioned only once” — because the default behaviour is to lose them.

Be explicit about coverage

The honest requirement, and the one most often skipped: say what was actually looked at.

“Across the 214 tickets from April to June, the most common themes were…” is a different claim from “the most common themes were…”, and only the first is verifiable.

Report:

How many documents were in the set, and the filter that defined it.

Whether the whole set was processed or a sample. If sampled, say so and say how.

What was excluded and why — unparseable files, permission-filtered documents, items outside the range.

Whether counts are exact or approximate.

A summary that doesn’t state its coverage can’t be evaluated. And if your system silently falls back to top-k retrieval for a question like this, the answer is actively misleading — it reads as a survey of everything and describes five documents.

Cost and latency

This pattern is expensive in a way ordinary RAG isn’t. Two hundred documents in batches of ten is twenty calls plus a combine, and the input token count is the whole corpus subset rather than a handful of chunks.

Practical consequences:

Don’t do it synchronously for large sets. A user waiting on a query touching hundreds of documents is waiting too long. Make it an explicit job with a result, not a chat turn.

Cache aggressively. Batch summaries for a fixed set are reusable across questions, and often across days. If your corpus is append-mostly, yesterday’s batch results are still valid.

Precompute where the questions are predictable. If the same weekly summary is wanted every Monday, compute it on Sunday.

Consider whether a query is the right tool. “How many tickets mention exports?” may be a database query with a text filter, answered exactly, instantly, and cheaply. Reaching for a language model to count things a query can count is a common and expensive mistake.

The check that matters

Spot-verify against ground truth you can compute independently. If the answer says forty-three tickets mention exports, grep for it. Being wrong on a countable claim is the failure that most damages trust in this kind of output, and it’s the easiest to test.

Then keep a couple of those checks as regression tests, because this pipeline has many stages and quiet degradation in any of them is hard to notice from the final answer alone.