AI quality is a release criterion, not a demo impression. Offline datasets measure reproducible behavior before deployment; online telemetry and reviewed samples detect drift and real-world failure. Safety combines deterministic controls, model-based checks, red-team cases, human oversight, and incident response. No single judge model or aggregate score proves a system safe.
Use production-inspired cases only after redaction, consent, access review, and sampling controls. Include common, long-tail, multilingual, ambiguous, adversarial, unanswerable, stale, cross-tenant, and dependency-failure cases. Freeze a holdout set to reduce overfitting.
Report slices, not just averages. A strong overall score can hide failure for a language, tenant policy, or rare high-risk action.
Use rubric-based model judges for scale only after calibrating them against multiple human reviewers. Randomize candidate order, separate the judge provider/model where useful, preserve rationales, and periodically re-check disagreement. Deterministic assertions should grade schemas, citations, access, tool permissions, and exact invariants.
Compare candidate versus baseline on the same dataset and runtime budgets. Define hard safety invariants, minimum slice quality, maximum regressions, latency/cost ceilings, and manual review for high-risk differences. Canary a small percentage, monitor, and retain immediate rollback to the previous prompt/model/index versions.
AI behavior drifts with prompts, models, retrieval data, policies, and traffic. The evaluation system is part of the backend: datasets, runners, traces, review queues, release records, and rollback are production capabilities.
Create a 30-case evaluation specification for classification plus grounded reply drafting. Include at least five adversarial/cross-tenant cases and five valid abstentions.
Acceptance criteria: hard authorization and citation invariants are deterministic, slices have thresholds, judge agreement is calibrated, a deliberately worse prompt fails the gate, and rollback is rehearsed.