Structured output from retrieved context

Sometimes the output of a RAG pipeline isn’t an answer for a human — it’s fields for a system. Pull the effective date, the limit, the responsible team out of these documents and return them as data.

This is easier than prose generation in one important way: a schema can be validated mechanically. And it introduces a problem prose doesn’t have, which is that every field needs a way to say “not present”, and models are reluctant to leave things blank.

Define the schema, use it

Don’t ask for JSON in the prompt and hope. Use whatever schema-enforcement your model provider offers — structured output modes, function/tool schemas, constrained decoding. These make malformed output largely impossible rather than merely unlikely.

Where that isn’t available, validate against a schema on receipt and retry on failure. Either way, never accept unvalidated structured output. It will be fine for a long time and then it won’t, and the failure will surface somewhere far from here.

Every field needs a null

The central discipline. Ask for effective_date and a model will supply a date. If the documents don’t contain one, it may supply a plausible one anyway — because the schema asked for a date and the schema is a strong instruction.

So:

Make fields nullable, explicitly, and say what null means.

Instruct on it directly:

Use null for any field the sources do not state. Do not infer,
estimate, or use a default. A null is a correct answer when the
information is absent.

Distinguish absent from empty. “No exceptions apply” and “the document doesn’t say whether exceptions apply” are different, and if both become an empty list you’ve lost the distinction. Two fields, or an explicit enum, handles it.

The failure this prevents is the worst one available in extraction: quiet invention of a value that then flows into a system as fact, with no prose around it to hedge and no reader to be sceptical.

Provenance per field

The structured-output equivalent of citation, and worth building in from the start:

{
  "carry_over_limit_days": {
    "value": 10,
    "source": 2,
    "quote": "Employees may carry over up to ten unused days."
  }
}

Verbose, and it gives you per-field verification: does the quote appear in the cited chunk, and is the value consistent with it? Both checkable mechanically.

If per-field provenance is too heavy, at minimum record which chunks the extraction was performed over, so a wrong value can be traced back.

Types earn their keep here

Prose can be vague; a typed field can’t be, which is an advantage.

Dates as ISO strings, not free text. “Next January” is not a date.

Numbers as numbers, with the unit in the field name — limit_days, not limit. Unit confusion in extracted data is a classic and silent error.

Enums for anything categorical, with an explicit “unknown” member. An enum without an escape hatch forces a wrong choice.

Booleans only for genuine binaries. Most apparent booleans are three-valued: yes, no, not stated.

Arrays for repeated items, with an instruction about ordering and whether the list is exhaustive.

Validate beyond the schema

Schema conformance is necessary and not sufficient. A well-formed object can be nonsense.

Range and sanity checks. A carry-over limit of 4,000 days is well-typed and wrong.

Cross-field consistency. An end date before a start date.

Quote verification, where you collected quotes. The strongest check available.

Coverage checks. If a field is null on every document, either the field isn’t in your corpus or the extraction prompt is wrong. Both worth knowing, and only visible in aggregate.

Extraction versus answering

Two things worth keeping distinct.

Extraction pulls stated facts out of documents. Low temperature, quote-verifiable, mechanically checkable. This is the reliable case.

Structured answering puts a judgement into a schema — a classification, a risk level, a summary field. The schema makes it look as reliable as extraction and it isn’t; the structure gives false confidence to what is still a model judgement.

Don’t mix them in one object without distinguishing them. If a response contains both extracted facts and inferred judgements, mark which is which, because downstream systems will otherwise treat them identically.

Multiple documents

When extracting one record across several chunks:

Say what to do about conflict. Two documents with different values need an explicit rule — most recent, most authoritative, or return both and flag it. Silent selection is the same problem as in prose answers, with less visibility.

Say whether to merge or list. Should three documents produce one record or three? Obvious to you, ambiguous in the prompt.

Keep per-record provenance when producing several, so each can be traced.

When to prefer prose

Structured output isn’t always the right target, and it’s easy to over-apply:

When the answer needs nuance. Conditions, exceptions and scope don’t fit in a scalar, and forcing them into one discards them.

When absence is common. A schema of mostly-null fields is a sign the corpus doesn’t contain what you’re asking for, and prose would have said so more usefully.

When a human is the consumer. Structure for machines, prose for people. A user reading a JSON object is a design failure somewhere upstream.

The pattern that works well: extract the checkable facts into a schema, generate the prose separately, and have the prose cite the same sources. Then the data is verifiable and the human-facing answer is readable, and both are grounded in the same retrieved material.