CI/CD, Migrations, Fixtures, and Retrieval Regression Tests
CI/CD for AI retrieval must test more than code. Start an ephemeral compatible PostgreSQL with pgvector, apply every migration from empty and from the prior release, load deterministic security fixtures, run SQL and API tests, validate plans, and compare golden retrieval metrics. Production delivery then uses backups, expand–migrate–contract, canaries, observability, and rehearsed rollback.
What will you be able to do?
- Build layered pipeline gates.
- Test migrations forward and backward where supported.
- Use deterministic and model-backed fixtures appropriately.
- Set quality, security, and performance thresholds.
- Promote the same immutable artifact through environments.
What are the pipeline layers?
- Formatting, lint, types, unit tests, secret/dependency scan.
- Ephemeral PostgreSQL + exact pgvector version.
- Migrate from empty; validate schema and extension.
- Upgrade a previous-version snapshot.
- Load two-tenant deterministic fixtures.
- Run CRUD, RLS, lifecycle, adapter, and API tests.
- Run exact/ANN and golden retrieval regression.
- Build immutable application artifact and provenance.
- Deploy to staging; run load, failover, and smoke tests.
- Canary production with automated stop/rollback gates.
Do not call a paid model in every unit test. Use deterministic fakes for logic and a controlled scheduled integration suite for real model behavior.
What are good gates?
- zero cross-tenant results;
- migrations complete within lock/time budget;
- required indexes valid and plans match allowed patterns;
- retrieval metrics do not regress beyond reviewed tolerances;
- p95 latency and error rate meet budget;
- deletion and rollback tests pass;
- artifacts, schema, model, and config versions are traceable.
How do you verify it?
Break the pipeline intentionally: wrong operator class, missing tenant predicate, invalid migration, dimension mismatch, reduced ANN effort, failed index build, and stale model. Each defect should fail a named gate before production.
What breaks in production?
- CI uses a different PostgreSQL/pgvector version.
- Migrations are tested only on empty databases.
- Snapshot fixtures contain personal data.
- ANN tests are nondeterministic without tolerances and frozen inputs.
- Rollback code is untested and incompatible with contracted schema.
AI pair-work prompt
Design a CI/CD pipeline for this pgvector repository [stack and deployment]. Include exact versions, empty/upgrade migrations, security fixtures, adapter/API tests, exact/ANN regressions, plan assertions, controlled model tests, artifact provenance, canary gates, and rollback drills.
Verification contract: Seed one known defect per gate and demonstrate that the pipeline blocks it with an actionable message.
Check your understanding
- Why test upgrades from a previous snapshot?
- Which tests need real embeddings and which do not?
- Define one retrieval regression threshold and its review process.
Official references