Observability

Why

Logs show events; observability helps explain behavior. AI systems need operational signals and semantic evidence connected in one trace.

Mental Model

Observe the decision path, not only the request path. A useful trace connects user intent, evidence, model behavior, tool actions, validation, feedback, cost, and latency.

Core Concepts

• Traces, spans, metrics, and structured events
• Prompt, model, schema, and dataset versions
• Token, latency, and cost attribution
• Semantic quality signals and user feedback
• Privacy-aware capture and retention
• Drift and cohort analysis

How It Works

A trace identifier is created at ingress and propagated through retrieval, model calls, tools, and validation. Each component emits structured attributes. Sensitive payloads are redacted or referenced securely. Aggregates lead back to representative traces.

In Harness

Shadow Harness is the observability layer: it records execution graphs, associates feedback with outcomes, and enables replay against candidate configurations.

Engineering

Define a stable event schema before dashboards. Record version identifiers, not ambiguous labels. Sample by risk as well as volume. Keep raw sensitive content separate from broadly accessible telemetry.

Trade-offs

Detailed traces accelerate diagnosis but increase cost and privacy exposure. Sampling reduces volume but may miss rare failures. Automated semantic signals scale but require calibration.

Common Mistakes

• Logging prompts without data classification.
• Aggregating failures before creating a taxonomy.
• Recording latency without stage attribution.
• Building dashboards that cannot link back to examples.

Practice

Instrument one end-to-end request. Starting from a bad answer, determine the evidence selected, model configuration, tool outcomes, validation decisions, total cost, and dominant latency without reading application code.

Sources

See the reference library for telemetry standards and production monitoring.