Shadow Harness

Engineering journal

A practical build log for an evaluation and observability layer that can capture production-shaped traces, replay tasks, compare candidates, and expose meaningful regressions.

CAPTUREinputs, versions, decisions, outcomes

REPLAYfrozen tasks against candidate systems

COMPAREquality, latency, cost, reliability

The problem

Model changes, prompt changes, retrieval changes, and orchestration changes interact. Production logs reveal real workload shape but often lack semantic structure. Offline suites are repeatable but drift away from reality. Shadow Harness joins the two without placing experimental output in the live response path.

System loop

Production-shaped task
        ↓
Sanitized trace capture
        ↓
Versioned replay queue
        ↓
Candidate execution
        ↓
Grading + operational metrics
        ↓
Slice comparison + error review
        ↓
Release evidence

Journal

Entry 01 · Trace schema before dashboard

The first artifact is an event contract: task identity, dataset version, configuration fingerprint, component spans, evidence references, output, grader result, timing, token use, and failure class. A dashboard built before this contract would freeze accidental field names into the product.

Entry 02 · Replay must declare side effects

Captured tool calls cannot be replayed blindly. Each tool is classified as pure, simulated, sandboxed, or prohibited. Replays consume recorded outputs by default; live re-execution is an explicit capability.

Entry 03 · Comparisons need paired cases

Candidate and baseline run on the same case versions. Paired outcomes make regressions visible with fewer examples and prevent workload changes from masquerading as system improvements.

Entry 04 · Aggregates link back to evidence

Every chart point must open the underlying cases. The system is designed for error analysis first and score reporting second.

Current milestone

DESIGN COMPLETE · PROTOTYPE NEXT

The trace contract, replay safety model, comparison unit, and release evidence format are defined. The next slice is a local executor with deterministic fixtures and an append-only result store.

Design principles

• Capture identifiers and references by default; retain sensitive payloads only under explicit policy.
• Keep the live path independent from shadow execution.
• Version datasets, prompts, models, graders, tools, and schemas.
• Treat latency, cost, failure, and abstention as outcomes—not metadata.
• Preserve raw case-level evidence behind every aggregate.

Continue to the architecture.