Evals Are Production Tests

The Test That Traditional Tests Miss

The uncomfortable thing about agent work is that the visible artifact can look fine while the work was wrong. A pull request can compile. The tests can pass. The final Slack answer can sound reasonable. The assistant can still have trusted the wrong source, missed the risk-bearing edge case, mutated state without enough authority, or summarized a private detail into a public channel.

Traditional gates catch deterministic failures. We need those gates more, not less. Formatting, types, lint, unit tests, integration tests, vulnerability scans, preview environments, and human review still carry the baseline contract. But agentic work adds a second contract: judgment. The system did not merely execute a known function. It selected context, interpreted intent, chose tools, navigated uncertainty, and decided when to stop.

That judgment needs tests of its own. Not because evals are magic, and not because a model can grade every model. Evals matter because they make the implicit standards of a workflow explicit enough to run, review, and improve.

In the Constructured world, that means an eval is not just “did the assistant write the expected paragraph?” It might be “did Q treat web content as untrusted while researching a cairn?” or “did the PR routine inspect unresolved review threads before editing?” or “did the operations helper distinguish a doc-vault export from live cron configuration?” Those are production questions. They deserve production tests.

What an Eval Is Actually For

The word eval has been pulled in two directions. One direction is the public benchmark: SWE-bench, MMLU, HumanEval, FrontierMath, GDPval, and whatever replaces this week’s leaderboard by next quarter. Benchmarks are useful. They help us notice broad capability shifts and pick a starting model or scaffold. They are not enough to certify our workflow.

The other direction is contextual evaluation: a local test of whether an AI system behaves well inside a specific product, team, domain, and operating constraint. That is the one we can act on. OpenAI’s public framing is useful here: specify what good means, measure against real conditions, and improve from errors. LangSmith’s docs put the same idea into lifecycle terms: offline evals before deployment, online evals in production, and a feedback loop between the two.

For us, the unit under test is rarely “the model.” It is the whole agent system: prompt, model, tool contracts, retrieval rules, sandbox policy, command surface, memory, human confirmation, trace capture, and final delivery. Changing any one of those can make the behavior better or worse. A model upgrade can break a prompt. A better prompt can expose a weak tool schema. A new tool can make an old safety instruction insufficient. A routing change can send a safe task through a lane with the wrong authority.

This is why model-only comparisons become misleading quickly. A weaker model with a better harness can outperform a stronger model with vague tools and no trace discipline on the workflow that matters.

The purpose of an eval, then, is not to crown a winner. It is to answer a change-management question: can this version of the system handle the cases we already know matter, and what new failures did reality teach us this week?

Start With the Decision, Not the Model

The fastest way to write a weak eval is to begin with the model output. “Was the answer good?” is too broad. Good according to whom? Good for which risk? Good before or after tool use? Good even if the agent skipped the required source? Good even if it was right for the wrong reason?

Start with the decisions that matter. In an agent workflow, the failure usually enters through a decision point before it becomes visible in the final artifact. The agent classifies the task. It chooses whether to read internal docs or web sources first. It decides whether a source is authoritative. It decides whether a command is safe to run. It decides whether to ask the user, continue independently, or stop. It decides what evidence belongs in the final answer.

Those decisions become the eval surface. For a research-and-publish routine, examples might grade whether the agent selected the oldest actionable request from a queue, avoided private-channel material, treated web pages as untrusted, cited primary sources, and did not put arbitrary external URLs into source-backed frontmatter. For a PR-review routine, examples might grade whether it distinguished resolved from unresolved threads, edited only the relevant files, and ran the smallest useful verification command before reporting success.

flowchart TD
  A[Workflow goal] --> B[Decision points]
  B --> C[Examples and edge cases]
  C --> D[Graders and assertions]
  D --> E[Gate in CI or routine rollout]
  E --> F[Production traces]
  F --> C

The diagram is deliberately circular. Evals are not a document you write once. They are a living map of the decisions that have mattered so far.

The Smallest Useful Dataset

A team does not need a giant dataset to start. In fact, a giant unreviewed dataset can make the system look more mature while making it harder to change. The smallest useful eval set is the one a domain owner can still read, argue with, and update.

Start with 20 to 50 examples. Include the common happy paths because regressions there are embarrassing. Include recent production failures because they are already paid-for lessons. Include near misses because they reveal brittle boundaries. Include adversarial cases where untrusted text tries to steer the system. Include expensive edge cases, even if they are rare, because a low-frequency mistake can still dominate risk.

A good example should include the input, the expected behavior, the unacceptable behavior, and the reason the case exists. That last field matters. Six months from now, someone will ask why the eval is so picky about one phrase, one source order, or one confirmation step. The answer should be in the fixture, not in someone’s memory.

There is a useful discipline here: an eval suite should be boring enough that a human can audit it. If nobody can explain why a case exists, retire or rewrite it. If a case catches a real regression, keep it and annotate it. If production reveals a new pattern, add the smallest example that preserves the lesson.

Graders Need Their Own Trust Model

Some eval checks are mechanical. Did the final answer include a permalink? Did a markdown file contain the required frontmatter? Did the agent run the build? Did it call a forbidden tool? Did a patch touch files outside the owned area? These should be code assertions. Do not spend model tokens judging a fact the machine can check directly.

Other checks are semantic. Was the final note appropriately concise for a public channel? Did the article explain the tradeoff without overstating the source? Did the agent’s review identify the most severe issue first? Did it preserve the user’s intent while refusing an unsafe action? These are places where rubric-based human review or an LLM grader can help.

The trap is treating an LLM grader as a neutral judge. It is another probabilistic system, usually reading the same kind of ambiguous text as the original agent. It can be biased by phrasing. It can miss subtle policy failures. It can reward fluency. It can drift when the underlying model changes. A grader is useful when it is calibrated and audited; it is dangerous when it becomes invisible authority.

Pairwise graders can be especially useful for prompt changes because the question becomes narrower: did candidate B improve this workflow relative to candidate A under the same fixture?

A practical trust model looks like this: deterministic checks first, rubric graders second, human audit around the grader, and escalation rules for high-impact failures. If the evaluator says a production-facing message is safe but the deterministic privacy check finds a token, ID, or secret-shaped string, the deterministic check wins. If the rubric grader passes a case that a domain owner rejects, the rubric changes.

Offline Evals Catch Regressions; Online Evals Catch Reality

Offline evals are the familiar shape. They run before a change ships. We take a curated dataset, execute the candidate version, grade the outputs and trajectories, and compare the result to the current baseline. This is the right place to catch prompt regressions, model-upgrade surprises, schema changes, and new tool behavior that breaks known cases.

Online evals are different. They sample live runs, often without a perfect reference answer, and ask whether production behavior still matches expectations. That matters because agents meet cases nobody thought to put in the fixture. Real users phrase requests strangely. Source documents drift. APIs fail halfway through a workflow. A tool returns a surprising error. A benign-looking web page contains adversarial instructions. The offline suite only knows yesterday’s lessons.

The two modes should feed each other. When an online eval or human review catches a miss, reduce it to a fixture and add it to the offline suite. When an offline eval starts failing after a change, inspect the trace and decide whether the system regressed or the expectation is stale. Either outcome is useful. A stale eval is not an annoyance; it is a prompt to update the team’s encoded standard.

sequenceDiagram
  participant Dev as Candidate change
  participant Offline as Offline eval suite
  participant Prod as Production runs
  participant Review as Human or rubric review
  participant Dataset as Eval dataset
  Dev->>Offline: Run known cases
  Offline-->>Dev: Block or approve rollout
  Dev->>Prod: Ship behind policy
  Prod->>Review: Sample traces and outcomes
  Review->>Dataset: Add new failure modes
  Dataset->>Offline: Strengthen future gates

Trace Data Is the Raw Material

The final answer is often too late to grade the work. Imagine an agent publishes a correct-looking summary. Did it use the authoritative internal doc or a stale web post? Did it see an injection attempt and ignore it, or did it never inspect the risky source? Did it run the build before pushing, or did it merely claim that it did? Did it ask for approval before a side effect? The answer lives in the trace.

This is where evals connect directly to observability. A trace should capture the workflow identity, source reads, tool calls, command outcomes, policy gates, approvals, artifacts, and final delivery. The eval harness can then grade not just what the agent said, but what it did. For agentic operations, trajectory is part of correctness.

The security payoff is significant. Prompt injection defenses often depend on process: trusted-first reading order, content wrapping, hardcoded rules, least privilege, and explicit refusal to let untrusted content trigger actions. A final answer cannot prove those defenses held. A trace can show whether web content was read after the trusted outline, whether suspicious instructions were flagged, whether commands came only from the operator’s plan, and whether final text avoided sensitive identifiers.

This does not mean broad transcript indexing. The trace can store references, hashes, redacted attributes, and access-controlled payload links. Evals need evidence, not a new leak surface.

Put Evals Beside the Existing Gates

A mature eval program should feel like an extension of the existing delivery pipeline, not a separate research ritual. We already know how to use gates: fast checks on every change, slower checks before merge or rollout, human review for judgment, and production monitoring after release. Evals slot into that pattern.

For prompt, tool, routing, or model changes, run a small fast suite automatically. It should be cheap enough that people do not avoid it. For changes that affect authority, public output, private data, or production mutation, run the heavier suite and require review of failures. For recurring autonomous routines, run canary sampling in production and keep a dashboard or report of failure classes over time.

The gate should not be a single magic score. Scores are useful, but failure class matters more. A one-point drop caused by a harmless formatting preference is not equivalent to one missed approval before a mutating tool call. Treat eval output like test output: inspect the failing cases, classify the risk, and decide whether to block, fix, or update the expected behavior.

eval_gate:
  fast:
    runs_on: [prompt_change, tool_schema_change, model_change]
    blocks_on: [policy_violation, missing_required_source, forbidden_tool_call]
  release:
    runs_on: [routine_rollout, authority_change]
    requires_review_for: [score_drop, new_failure_class, grader_disagreement]
  production:
    samples: traces
    feeds_back_into: offline_dataset

What to Remember

The teams that get reliable value from agents will not be the teams with the fanciest demo prompt. They will be the teams that turn their local taste into executable checks: examples, rubrics, traces, gates, reviews, and feedback loops. That is slower than hoping the next model fixes everything. It is also how engineering usually wins.

The public benchmark still matters. SWE-bench and similar harnesses show that agent evaluation needs real tasks, codebases, and executable outcomes. NIST’s AI risk work reminds us that evaluation belongs across the lifecycle, not just at launch. Product eval platforms give us vocabulary for offline and online measurement. But the durable advantage is local: our workflows, our edge cases, our data boundaries, our users, our notion of a good result.

For Constructured, the useful next step is modest. Pick one recurring agent lane, probably one with clear authority boundaries and repeated structure. Write down the decisions that matter. Build 20 examples. Add deterministic assertions for the facts. Use a rubric grader only where judgment is real. Sample production traces. Every time the lane surprises us, add the smallest fixture that would have caught it.

1. Test the judgment layer - Traditional checks prove the artifact; evals prove the agent's decisions, trajectory, and communication.
2. Keep evals local - Public benchmarks help choose tools, but production evals encode our workflows and risks.
3. Grade traces, not just answers - Source order, tool calls, approvals, policy boundaries, and command results are part of correctness.
4. Close the loop - Offline evals catch known regressions; online evals discover new cases; both should update the dataset.
5. Audit the graders - Deterministic checks should win where facts are mechanical, and humans should calibrate semantic rubrics.

Discussion Prompts

• Which recurring agent lane should get the first 20-case eval suite: Cairns publishing, GitHub PR review, issue triage, or OpenClaw maintenance?
• What are the three highest-impact decisions in that lane that a final answer alone cannot prove?
• Which production traces already contain enough evidence to create eval fixtures, and which missing fields would make grading possible?

References

1. OpenAI: How evals drive the next chapter in AI for businesses - Defines contextual evals as a Specify, Measure, Improve loop and emphasizes domain-expert ownership.
2. OpenAI API: Evaluation best practices - Current OpenAI API guidance for designing evals, useful even as the older Evals platform moves through deprecation.
3. OpenAI API: Deprecations - Notes the June 3, 2026 deprecation timeline for the Evals platform; a reminder to separate eval discipline from any one hosted product.
4. LangSmith: Evaluation concepts - Explains evaluator types and the offline/online split in a way that maps cleanly to agent delivery pipelines.
5. NIST AI Risk Management Framework - Places evaluation inside AI lifecycle risk management, including generative AI profile work and critical-infrastructure direction.
6. SWE-bench leaderboards - Public coding-agent benchmark family showing why realistic tasks, harnesses, and solved-instance metrics matter.
7. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? - Research arguing that harder, contamination-resistant, long-horizon software tasks are needed as simpler coding benchmarks saturate.
8. Trace the Work, Not Just the Error - The local Cairns companion on why agent operations need causal traces before they can be evaluated well.