Incremental graph refresh and staleness control

Mutating local subgraphs dynamically to keep pace with document modifications.

Why it matters

Triggering a multi-thousand-dollar global graph rebuild because a single source document was updated or deleted is an engineering failure. Incremental refresh isolates data modifications, applying targeted mutations only to impacted nodes, edges, and community summaries to guarantee real-time data freshness with near-zero overhead.

Source data changes daily; rebuilding the entire graph for one updated document is an engineering failure. Incremental refresh applies targeted mutations only to the impacted subgraph.

refresh_pipeline:
  source_event: policy_updated
  impacted_entities: { strategy: 'entity index by doc_id', expected: ['Policy P-12', 'RefundRule'] }
  tasks: [re-extract changed doc, re-link aliases, invalidate impacted community summaries]
  sla: { max_staleness_minutes: 20 }

The pipeline has three moving parts: (1) map the changed document to the entities it touches via a doc→entity index, (2) re-extract and re-canonicalise just those entities/edges, (3) invalidate the community summaries built on them — a forgotten summary is how stale answers survive even after the underlying chunk was fixed.

Freshness is an SLA, not a vibe. max_staleness_minutes makes 'how current is the graph?' measurable and alertable. Tightening it costs infrastructure (more frequent re-extraction), so it's a deliberate trade, not a default.

Plan for failed refreshes. A refresh event can fail mid-way, leaving the graph partially updated. You need a backfill/replay step driven by the event log so a dropped event doesn't silently leave a pocket of stale data — and monitoring for refresh skew between source events and what the index actually reflects.

Incremental refresh is restocking only the sold-out shelves, not re-buying the whole supermarket every night. And the receipt log (event log) lets you replay any delivery that the truck dropped, so no shelf is silently left empty.

• Full rebuild per change. Symptom: cost + downtime. Fix: subgraph-scoped mutation.
• Forgetting to invalidate summaries. Symptom: stale answers after a fix. Fix: invalidate impacted community summaries.
• No backfill for failed events. Symptom: silent stale pockets. Fix: replay from event log.
• No freshness SLA. Fix: set and alarm on max_staleness_minutes and refresh skew.