Exact Search as the Quality Baseline

Before tuning HNSW or IVFFlat, capture exact top-k results for a versioned dataset and query set. Exact pgvector search establishes the nearest stored rows under one metric and filter. Comparing ANN results with this baseline measures approximation recall; comparing either result with human relevance labels measures product quality. Those are different questions.

What will you be able to do?

Freeze a reproducible retrieval experiment.
Save exact filtered top-k results.
Calculate ANN overlap and approximation recall.
Keep human relevance separate from geometric ground truth.
Identify slice-specific quality failures.

What must an experiment record?

Record dataset snapshot/checksum, query IDs and text, query embedding model, document model, metric, preprocessing, SQL, filters, top-k, database and extension versions, planner settings, code revision, timestamp, hardware class, and cache condition. Without these, “recall improved” is not reproducible.

How do you capture an exact run?

Use a dedicated evaluation transaction and prevent approximate index scans locally:

sql

BEGIN;
SET LOCAL enable_indexscan = off;

INSERT INTO eval.exact_results (run_id, query_id, rank, chunk_id, distance)
SELECT $1, $2, row_number() OVER (ORDER BY c.embedding <=> $3::vector),
       c.id, c.embedding <=> $3::vector
FROM app.chunks c
JOIN app.documents d ON d.id = c.document_id
WHERE d.tenant_id = $4 AND d.is_current
ORDER BY c.embedding <=> $3::vector
LIMIT $5;

ROLLBACK; -- Use COMMIT only in the evaluation database after inspection.

Do not disable planner features globally. On a read replica, confirm replication lag and dataset consistency before treating results as a baseline.

How do you compare ANN?

For each query, count exact top-k IDs present in ANN top-k. Average across queries, then break down by tenant, filter selectivity, language, source type, query class, and rare intents. Also inspect ranking order and distance distribution. High geometric recall can still retrieve irrelevant content if the embedding or chunks are poor.

How do you verify it?

Repeat the exact run and confirm identical ordered IDs for an unchanged snapshot. Introduce one known vector and predict where it ranks. Then create an intentionally low-quality ANN configuration and prove approximation recall falls while exact results remain stable.

What breaks in production?

The exact and ANN queries use different filters or top-k.
Evaluation data changes mid-run.
Only average recall is reported.
Exact geometric neighbors are called human relevance truth.
Planner changes or a hidden index invalidate the baseline.

AI pair-work prompt

Audit this retrieval experiment [paste config/results]. Identify missing reproducibility fields, differences between exact and ANN SQL, metric ambiguity, filter mismatches, and slices hidden by the average. Return corrected experiment metadata and assertions.

Verification contract: Another engineer must reproduce the exact ordered IDs from the saved snapshot and run definition.

Check your understanding

What does approximation recall measure?
How does it differ from task relevance?
List ten fields needed to reproduce an exact run.

Official references