AI evaluation is the release and operations discipline that measures whether a system is useful, grounded, safe, fair enough for its context, fast, and affordable. Test the complete application—including retrieval, prompts, tools, policies, and user interface—not only the base model. Trace every stage while minimizing sensitive data.
Method note: Evaluation design and red-team release gates require the same evidence regardless of the runner. Portal dashboards, SDK evaluators, and CI jobs can execute parts of the process, but they are complementary stages rather than interchangeable ways to perform one operation.
Keep a hidden holdout set. Version dataset, labels, evaluators, prompt, retrieval config, model deployment/version, and code commit.
Do not collapse everything into one score. A higher average cannot compensate for unauthorized data disclosure.
Use a clear rubric and examples, then calibrate evaluator scores against qualified humans. Measure agreement and review disagreements. Keep deterministic checks for JSON schema, citations, access filters, banned operations, and exact facts. A model judge can be biased by style, verbosity, and shared failure modes.
Microsoft Foundry evaluation tools can run built-in and custom evaluators. Treat their output as evidence within your release policy, not an automatic guarantee.
Trace request ID, authenticated principal category/tenant pseudonym, prompt version, model deployment/version, retrieval query and document IDs, tool calls and outcomes, latency, token usage, safety decisions, errors, and final status. Redact or hash sensitive content; configure retention and access.
OpenTelemetry connects application spans with Azure Monitor/Application Insights and supported Foundry tracing. Do not log access tokens, API keys, raw secrets, full protected documents, or unnecessary personal prompts.
Test categories rather than random clever prompts:
Test both success of defenses and graceful product behavior. Keep red-team data secured because it can contain exploit methods and sensitive examples.
A release policy might require:
Canary a new model or prompt on a small approved traffic slice, compare online metrics, and stop automatically on guardrail breach.
Document intended use, excluded use, users, affected people, failure impact, data sources, evaluation limits, oversight, accessibility, security, transparency, and appeal. Put owners and dates on risk controls. Show users when they interact with AI, cite sources where helpful, and provide a path to correct or contest consequential output.
Prepare to disable a model deployment, revoke tool access, block a prompt/version, isolate a tenant, preserve evidence, notify owners, correct data, replay safe work, and communicate impact. An AI incident joins—not replaces—the normal security, privacy, reliability, and customer-support process.