Events are durable facts about lifecycle; traces connect causal work; metrics summarize distributions; logs explain local diagnostics; replay reconstructs a run from versions, events, and sanitized fixtures. A production harness uses all five with shared identifiers. Observability must reveal tool and policy behavior without exporting secrets, hidden reasoning, or unnecessary personal data.
You will define a normalized event vocabulary, correlation model, redaction strategy, and replay package.
Each event needs run ID, sequence, timestamp, tenant-safe subject, provider, versions, and typed data. Avoid raw prompt/result content by default.
Use trace/span IDs on events and logs. Use low-cardinality labels on metrics; never put ticket IDs or prompts in metric labels.
Instruction/schema/tool/policy versions, route decision, sanitized input, context manifest, stubbed tool results, event stream, random/temperature settings where applicable, and expected assertions. Replaying against a live mutable tool is not reproducible and may cause side effects.
Failure injection: Seed canary PII and a fake secret across user input, retrieved context, and tool output. Inspect all five observability surfaces and exported provider traces.
Take a failure and answer: what task ran, which versions, which provider, what tools were proposed/allowed, what changed externally, why it stopped, what it cost, and which evaluation should catch it. If any answer requires guessing, add evidence—not more generic logs.
Define JSON schemas for ten normalized events and a reducer that builds a timeline. Add redaction and schema-compatibility tests.