Evaluation

Why

Without evaluation, development follows memorable anecdotes. Evaluation turns desired behavior into evidence that can guide design and releases.

Mental Model

An evaluation is a measurement system: task distribution + execution environment + grader + analysis. A score without coverage and error taxonomy is rarely actionable.

Core Concepts

• Representative and adversarial task sets
• Deterministic checks, model graders, and human review
• Pairwise comparison and blinded experiments
• Retrieval and end-to-end metrics
• Confidence intervals and slice analysis
• Regression gates and production feedback

How It Works

Versioned cases run against a candidate system under recorded settings. Graders produce outcomes and rationales. Results are aggregated by meaningful slices, compared with a baseline, and inspected through individual failures before a release decision.

In Harness

Shadow Harness links each score to its exact input, configuration, trace, output, grader version, and evidence. No aggregate is allowed to become detached from examples.

Engineering

Begin with 20 difficult real cases. Add every important production failure as a regression case. Calibrate model-based graders against periodic human labels. Track latency and cost beside quality.

Trade-offs

Deterministic graders are stable but narrow. Model graders scale semantic judgment but may share biases with the system. Large suites improve coverage while slowing iteration and increasing expense.

Common Mistakes

• Optimizing a single aggregate score.
• Allowing test cases to leak into prompts.
• Changing the grader during a candidate comparison.
• Ignoring abstention, safety, latency, and cost.

Practice

Define a task rubric with three quality dimensions and two operational constraints. Label 25 examples twice, resolve disagreements, and compare the rubric with an automated grader.

Sources

See the reference library for evaluation design and statistical practice.