Ingestion starts before embedding. A trustworthy extractor identifies the source, captures an immutable version, parses content and structure, records permissions, normalizes under a versioned policy, computes checksums, and saves evidence. If extraction silently drops tables, pages, or access labels, no embedding model or vector index can recover the missing truth.
Represent every extracted document with:
Keep the original source in its approved system of record. The retrieval database holds a rebuildable serving projection unless your architecture explicitly says otherwise.
Normalization may standardize line endings, Unicode, repeated whitespace, boilerplate, or HTML. It must not silently alter identifiers, negate meaning, merge table cells, or remove security labels. Version the policy and test it with golden input/output fixtures.
Parsing files is a security boundary. Enforce size and type limits, scan where policy requires, isolate parsers, reject encrypted or malformed files deliberately, and never execute macros or document code. Treat embedded instructions such as “ignore all previous rules” as content, not control.
Create golden fixtures for PDF, HTML, Markdown, a table, code, Unicode, an empty file, a malformed file, and a restricted document. Assert extracted structure, exact checksums, warnings, and permissions. Compare source inventory with database inventory and account for every missing, failed, current, or deleted source.
Create an extraction test plan for these source types [list]. Include stable identity, version detection, structural preservation, permissions, Unicode, tables, malformed files, parser isolation, checksums, and reconciliation. Separate content warnings from fatal failures.
Verification contract: Every fixture must have deterministic expected output and provenance. No document becomes retrievable until its access metadata and normalized checksum are present.