Trace, Debug, and Evaluate OpenAI Agent Workflows

Tracing explains one run by recording model calls, tools, handoffs, guardrails, and custom spans. Evaluation measures behavior across a representative task suite. Use traces to discover failure mechanisms, turn those mechanisms into scored cases, compare against a baseline, and block releases that regress quality, safety, latency, or cost.

What will you be able to do?

You will group related runs in a trace, define a multi-dimensional evaluation scorecard, and convert a production failure into a regression test.

How do you group a workflow trace?

trace_demo.py:

python

import asyncio

from agents import Agent, Runner, trace


triage = Agent(
    name="Support triage",
    instructions="Classify the request and state whether human review is needed.",
)


async def main() -> None:
    with trace("SupportOps evaluation sample"):
        first = await Runner.run(triage, "My invoice total looks wrong.")
        second = await Runner.run(
            triage,
            f"Review this draft classification for policy risk: {first.final_output}",
        )
        print(second.final_output)


if __name__ == "__main__":
    asyncio.run(main())

Built-in tracing is normally enabled on the server-side SDK path. Review data governance before sending sensitive inputs to any observability system, and use SDK controls where tracing must be reduced or disabled.

What should an evaluation score?

Keep dimensions separate:

Dimension	Example assertion
Task quality	category matches labeled intent
Tool behavior	correct tool and arguments
Grounding	every policy claim has valid source
Safety	mutation never occurs without approval
Stop behavior	loop ends within budget
Latency	p95 meets the service objective
Cost	usage proxy remains below threshold

An average score must not hide a safety failure. Use hard gates for critical invariants.

How do you build a useful dataset?

Start with real task shapes: normal, boundary, ambiguous, adversarial, provider-error, and historical incident cases. Remove secrets and personal data while preserving the failure mechanism. Version cases, rubrics, judge prompts, and expected tool/policy events.

Use deterministic graders whenever possible. Use model graders for semantic qualities with a specific rubric and calibration against human labels. Never let the same generated reference leak the answer into the candidate context.

Failure injection: Change the billing specialist’s handoff description so cancellation requests route incorrectly. A routing evaluation should fail before deployment; a trace should show why.

What changes in production?

Sample and retain traces by risk, connect trace IDs to business outcomes, monitor distributions rather than anecdotes, and require baseline comparison in CI. Separate online monitoring from offline release evaluation.

What mistakes should you avoid?

Treating trace presence as evaluation.
Grading only final prose.
Using five happy-path prompts as a benchmark.
Changing dataset and candidate simultaneously.
Ignoring variance and non-determinism.

Check your understanding

What question does a trace answer versus an evaluation?
Which requirements should be hard release gates?
How does an incident become a regression test safely?

Expert extension

Create a 30-case SupportOps suite and a scorecard with hard safety gates. Run multiple samples for ambiguous cases and report variance, not only a mean.

Official references