Reliability

Why

AI systems inherit ordinary distributed-system failures and add semantic failures that still return HTTP 200. Reliability must cover both transport and meaning.

Mental Model

Reliability is controlled degradation. A robust system knows its budgets, detects uncertainty, contains failure, and preserves a useful path when ideal behavior is unavailable.

Core Concepts

• Timeouts, retries, backoff, and circuit breakers
• Idempotency, queues, and dead-letter handling
• Rate, token, latency, and monetary budgets
• Fallbacks, abstention, and human escalation
• Schema validation and policy enforcement
• Blast-radius reduction

How It Works

Requests move through bounded stages. Each stage has a deadline, contract, telemetry, and defined failure behavior. Retries target transient errors only. Semantic checks route uncertain outputs to repair, fallback, or abstention.

In Harness

The harness classifies failures by layer and decision. It exposes retry amplification, fallback frequency, validation failures, and budget exhaustion as first-class signals.

Engineering

Set an end-to-end deadline and allocate it across stages. Make state-changing operations idempotent. Use exponential backoff with jitter. Test partial dependency loss. Document what users experience during degradation.

Trade-offs

Retries improve transient success but amplify load and cost. Strict validation prevents bad outputs but may reduce completion rate. Fallback models preserve availability with different behavioral characteristics.

Common Mistakes

• Retrying every error with identical inputs.
• Giving each nested call the full request timeout.
• Hiding degraded behavior from telemetry.
• Measuring availability without correctness.

Practice

Draw a failure tree for a retrieval-assisted request. Inject one failure at each boundary and confirm the observed behavior, trace, user message, and recovery path.

Sources

See the reference library for resilience and site-reliability patterns.