Choose an embedding model by testing it on your users, languages, content types, query styles, privacy rules, latency, throughput, dimensions, and budget. Public benchmarks are useful filters, not deployment evidence. Build a labeled query set, compare candidates under the same retrieval pipeline, inspect failures, and keep a reversible versioned migration path.
Do not begin with “which model is best?” Begin with “what failures are unacceptable for this workload?”
Collect 50–200 real or carefully reviewed queries across:
For each query, label relevant chunk IDs and optionally grades such as 0–3. Keep the evaluation outside the retrieval code so candidate models cannot influence labels.
Use the development split to choose chunking, model, and parameters. Report final results on a held-out test split to reduce tuning bias.
Start with recall@k: how often the retrieved top-k contains labeled relevant material. Add precision@k, mean reciprocal rank, and nDCG for graded ranking. Record p50/p95 latency, throughput, vector size, database size, provider cost, and error rate. Later, measure RAG groundedness separately; better answer wording does not prove better retrieval.
Write an architecture decision record containing candidates, dataset, metric definitions, results with confidence limits when possible, failure slices, privacy review, cost assumptions, chosen model and revision, alternatives rejected, and triggers to reevaluate. Keep raw run artifacts so another engineer can reproduce the table.
Blind the model names in the final quality review if feasible. Run each candidate with identical chunks, filters, top-k, and exact search first. Manually inspect at least the worst ten queries and every security-sensitive query. Repeat a sample run to detect nondeterministic preprocessing or provider changes.
Convert this product description [paste] into an embedding-model evaluation plan. Produce a scorecard, representative query slices, labeling guide, metrics, privacy questions, cost inputs, and decision thresholds. Do not name a winner without results. Identify where human relevance judgment is required.
Verification contract: A reviewer must be able to reproduce every reported number from saved queries, labels, model identity, embeddings, retrieval configuration, and code revision.