Observability with Logs, Metrics, Traces, and AI Signals

Observability lets operators ask new questions about a running system using logs, metrics, traces, profiles, and domain signals. Instrument boundaries rather than every line, propagate trace context across HTTP and jobs, control sensitive content and cardinality, and connect technical health with AI quality, safety, latency, and cost service levels.

What will you be able to do?

• define useful events, metrics, spans, and service-level indicators;
• trace one AI run across API, queue, retrieval, provider, and tools;
• avoid high-cardinality and sensitive telemetry;
• alert on user impact rather than raw noise.

Which signal answers which question?

• Metrics: Is error rate or latency changing across the fleet?
• Traces: Where did one distributed request spend time or fail?
• Logs: What discrete event and diagnostic context occurred?
• Profiles: Which code consumes CPU or memory over time?
• Evaluations: Was the AI result correct, grounded, safe, and useful?

Instrument spans such as http.request, db.query, queue.publish, job.execute, retrieval.search, model.generate, tool.execute, and evaluation.grade. Attach bounded attributes: route template, feature, outcome category, model alias, prompt/schema version, retry count, token usage, and safe tenant tier—not raw prompts or unique ticket IDs as metric labels.

What should the service-level objectives measure?

Examples require product-specific targets, not copied numbers:

• successful authorized ticket reads within a latency threshold;
• AI draft operations complete or return a safe terminal outcome within a time window;
• citation correctness and groundedness above release/production thresholds;
• unauthorized tool effect rate equals zero;
• budget reconciliation and evaluation pipeline freshness.

Use latency histograms and percentiles; averages hide tails. Define what counts as valid traffic and how dependency-caused failures affect the indicator.

How should sensitive AI telemetry be handled?

Default to metadata and references. If sampled content is necessary for debugging or evaluation, apply consent, redaction, encryption, access control, regional policy, limited retention, audit, and deletion. Separate operational logs from governed evaluation stores.

Why does this matter in AI-native systems?

Traditional 200 OK can hide hallucination, unsupported citations, or a silently degraded fallback. Correlate runtime traces with AI-run records and post-hoc evaluation. Monitor model/provider/version mix so routing changes are visible.

Common mistakes

• Every user or ticket becomes a metric label.
• Auto-instrumentation captures request bodies or authorization headers.
• Sampling drops every rare error trace.
• Trace context is lost when publishing a job.
• Alerts fire on CPU alone with no user impact.
• Dashboards show model latency but not quality, cost, refusal, or safety.

AI Pair-Programmer Prompt

Exercise

Draw and instrument the trace for create-AI-draft from HTTP through queue, retrieval, model, and saved result. Inject a slow query and provider timeout.

Acceptance criteria: one trace locates each delay, metric labels stay bounded, logs carry correlation without private content, and alerts distinguish database latency from provider failure.

Knowledge check

1. Can HTTP success prove AI quality?
2. Why avoid ticket IDs in metric labels?
3. Where should trace context go for a job?

---

Answers

1. No; semantic evaluation is separate.
2. They create high cardinality, cost, and privacy exposure.
3. In standardized propagation metadata attached to the message or durable job context.

Completion checklist

• End-to-end traces cross asynchronous boundaries.
• SLOs reflect user and AI outcomes.
• Sensitive telemetry has explicit governance.

Primary references

• OpenTelemetry
• Prometheus metric practices
• Google SRE Workbook: Alerting on SLOs