Agent evaluation is executable requirements applied to final outcomes and intermediate behavior. A strong suite combines deterministic domain tests, provider adapter contracts, labeled task cases, rubric graders, adversarial security tests, and operational SLO checks. Releases compare a candidate to a fixed baseline and fail hard on critical safety or authority regressions.
You will create an evaluation pyramid, design a case schema, and define release gates that cannot be hidden by an average score.
Run cheap deterministic layers often. Use live-model evaluations for behavior that actually depends on models.
Keep graders blind to candidate identity. Calibrate model graders against human labels and monitor agreement.
Report per-category distributions and confidence/variance where sampling matters.
Failure injection: Improve the average by failing rare destructive-action cases. A hard safety gate must block release despite the higher mean.
Implement a provider-agnostic evaluator that consumes normalized events and outcomes, then compare OpenAI and Claude on the same SupportOps suite without requiring identical internal traces.