Provenance, citations, and claim support

Hard-wiring auditability by mapping every extracted node and edge back to the exact source text span it came from.

0/4 done

Overview

Hard-wiring auditability by mapping every extracted node and edge back to the exact source text span it came from.

Why it matters

A beautifully constructed graph is still a black box if a user has no way to check where an answer came from — and in enterprise settings (legal, financial, medical), an unverifiable answer is often worse than no answer at all. Provenance is the practice of attaching, to every node and edge in the graph, a pointer back to the original document, chunk, and text span it was extracted from — so nothing in the graph is 'just asserted', everything is traceable.

This pays off at generation time in the design of the citation, not just its presence. Compare two approaches to citing a multi-part answer:

  • One generic citation block after the whole answer ('Sources: [1][2][3]') tells the reader that something in the answer came from somewhere in those three documents — but if the answer has four claims and one of them is wrong, there's no way to isolate which source (if any) actually supports it.
  • Per-claim citations link each individual assertion in the answer to the specific graph edge(s) and source span(s) that support it. Now a user — or an automated evaluator — can check claim-by-claim whether the graph traversal actually justifies what the LLM said, and catch exactly where a hallucination slipped in.

Per-claim provenance is what turns GraphRAG from 'a fancier way to generate plausible text' into 'a system whose every output is independently checkable' — the property that makes it usable in regulated or high-stakes domains.

How it actually works

In a regulated or high-trust setting, an answer is only as good as its evidence trail. Professional GraphRAG attaches support to each claim, not one citation block bolted onto the end.

{
  "answer": "Alice approved the exception under Policy P-12.",
  "claims": [
    {"text": "Alice approved the exception",
     "support": {"graph_edges": ["Alice-approved-Exception42"], "chunks": ["doc://approvals/42#L10-18"]}},
    {"text": "Exception42 is governed by P-12",
     "support": {"graph_edges": ["Exception42-governedBy-P12"], "chunks": ["doc://policy/p12#L3-11"]}}
  ]
}

Why per-claim, not per-answer. A single citation block lets one unsupported sentence hide among three supported ones. Per-claim support makes every sentence independently verifiable — and makes the unsupported claim stick out, so a critic node (or a human) can reject exactly it instead of the whole answer.

Store it as structured fields, not markdown. claim_id, source_id, edge_path, timestamp are queryable; a markdown footnote string is not. Structured provenance lets you audit ('show me every answer that cited the now-retracted doc'), prefer the newest valid evidence, and prove compliance.

Pair it with a refusal contract. If a claim has no support, the system must say so or refuse — provenance is what makes 'I don't have evidence for that' a deterministic behaviour rather than a hope.

Analogy

Per-claim citations are receipts for each line item, not one lump total at the bottom of the bill. With line-item receipts an auditor can challenge a single charge; with only a total, one padded charge hides inside a plausible sum.

Pitfalls & how to avoid them

  • One citation block per answer. Symptom: unsupported sentence hides. Fix: support per claim.
  • Citations as markdown text. Symptom: not auditable/queryable. Fix: structured fields.
  • No refusal when support is missing. Symptom: confident gaps. Fix: explicit 'insufficient evidence' path.
  • Stale evidence preferred. Fix: attach timestamps; prefer newest valid source.

Apply it to your system

Audit one real answer your system produced.

  • Can every sentence be traced to a specific edge or source span today?
  • Which provenance fields are missing for a real audit?
  • What should the system do when a claim has no supporting evidence?

Reading in progress · 0 of 4 activities done