LLM Systems

Why

Language models are probabilistic components, not deterministic databases. Useful systems emerge when their uncertainty is constrained by context, interfaces, validation, and feedback.

Mental Model

Think of a model call as a compiled execution request: instructions and context form the program, decoding is execution, and the response is an untrusted result that must cross a boundary before use.

Core Concepts

• Tokenization: the model operates on token sequences, so cost and context are measured in tokens rather than characters.
• Context: instructions, examples, retrieved evidence, tool results, and conversation state compete for finite attention.
• Decoding: temperature and sampling settings shape variation but do not create knowledge.
• Structured output: schemas narrow the output space and move validation into an explicit contract.
• Model routing: quality, latency, context capacity, and price are workload-specific decisions.

How It Works

Text becomes tokens, tokens become vectors, and repeated attention and transformation layers produce a probability distribution for the next token. Generation repeats until a stop condition. The surrounding application assembles context, calls the model, validates the result, and decides what happens next.

In Harness

Shadow Harness records the full request envelope, model settings, schema version, response, latency, and token use. That makes behavior reproducible enough for comparison even when individual outputs vary.

Engineering

Version prompts as code. Separate stable instructions from volatile evidence. Use typed schemas at component boundaries. Set timeouts and output limits. Capture a trace identifier before the call and propagate it through downstream work.

Trade-offs

Larger models often improve difficult reasoning but increase latency and cost. Longer context can add evidence while diluting critical instructions. More examples can improve format consistency while reducing room for task data.

Common Mistakes

• Treating a fluent response as a verified fact.
• Mixing policy, task, evidence, and user content without clear boundaries.
• Changing model and prompt simultaneously, making regressions hard to attribute.
• Parsing prose with fragile regular expressions when a schema is available.

Practice

Build one task with a typed response. Create five adversarial inputs, measure schema success, then compare a short prompt with an example-heavy prompt. Record quality, latency, and token cost.

Sources

See the reference library for transformer, model, and structured-output documentation.