Change Detection, Re-embedding, Tombstones, and Deletion

Content and permissions change continuously. A correct pipeline detects source versions or checksums, rebuilds only affected chunks, publishes a complete new version atomically, and hides stale versions. Deletion must propagate through chunks, embeddings, indexes, caches, answer logs, and citations according to policy. Periodic reconciliation catches events that webhooks or queues missed.

What will you be able to do?

• Detect meaningful source changes.
• Publish new document versions without partial visibility.
• Propagate access revocation quickly.
• Use tombstones to reconcile deletion.
• Prove stale content is not served.

What counts as a change?

Track source-native version/ETag plus raw and normalized checksums. A source timestamp alone may be noisy or unreliable. Parser, normalization, chunker, model, and permission changes can also require derived-state work even when source bytes stay constant.

How do you switch versions safely?

Build the new document version, chunks, permissions, and embeddings in a non-visible state. Verify counts and quality, then update the visible version pointer or status in one short transaction. Retire the old version after caches and readers honor the cutover.

Use a partial unique constraint to ensure only one current version per tenant/source.

What does deletion require?

A tombstone records that a known source was deleted or access revoked, even if physical cleanup is asynchronous. Retrieval excludes tombstoned rows immediately. A deletion worker removes derived data and invalidates caches. Retention, legal hold, and audit policy decide whether and when original records or logs are physically erased.

How do you verify it?

Automate a lifecycle test: create, retrieve, modify, cut over, revoke access, delete, and attempt old citation resolution. Search cache keys and logs under approved test IDs. Reconcile source inventory and database inventory until every source is current, pending, failed, or tombstoned.

What breaks in production?

• Old and new chunks are visible together.
• Access revocation waits behind a low-priority embedding queue.
• A webhook deletion is lost.
• Caches omit source version from their key.
• Audit retention conflicts with deletion policy and nobody owns the decision.

AI pair-work prompt

Model the complete lifecycle for this source connector [describe]. Include version detection, derived-state dependencies, atomic visibility, permission-only changes, tombstones, retention/legal hold, cache invalidation, reconciliation, and proof-of-deletion tests.

Verification contract: A fixture's old content must become unreachable through semantic, lexical, cache, citation, export, and agent-memory paths within the defined revocation SLO.

Check your understanding

1. Why is modified time alone weak change evidence?
2. What should happen before a new version becomes visible?
3. Which paths must honor a tombstone?

Official references

• PostgreSQL partial indexes
• PostgreSQL constraints