Traditional software has a contract: given this input, return that output. Agents do not offer exact outputs, but enterprises still need a contract. Evaluations are how you write one.
Three layers of evaluation
- Unit evals check a single capability: does the extractor return valid JSON for these hundred invoices?
- Trajectory evals check the path: did the agent call the right tools in a reasonable order, and stop when it should?
- Outcome evals check the business result: was the ticket resolved correctly, the record updated accurately, the policy respected?
Most teams start and stop at outcomes. Trajectories are where you find the bugs that happen to succeed this time.
Build the set from reality
The most valuable eval cases come from production: real inputs, real edge cases, real failures. Make capturing them frictionless. Every escalation, correction, or complaint becomes a candidate case. A small, curated set of a few hundred honest cases beats thousands of synthetic ones.
Grading, honestly
- Deterministic checks first. Schema validity, required fields, forbidden actions, numeric tolerances. They are cheap and unambiguous.
- Model-based graders second. Useful for tone, completeness, and faithfulness to sources. Calibrate them against human labels and re-check periodically, because graders drift too.
- Human review for the hard tail. Sample, do not skip.
Track pass rate *per category*, not only overall. An average of ninety percent can hide a category at fifty that matters most.
Safety evals are not optional
Include adversarial cases: prompt injection in retrieved content, requests to exceed authority, attempts to extract other tenants' data. Define the expected behavior as a refusal or escalation, and fail the build when it regresses.
Wire evals into change control
Any change to prompt, model, tool schema, or retrieval configuration runs the suite. Regressions beyond a threshold block the release, the same way failing unit tests do. This is what makes it safe to swap models when a better or cheaper one arrives, instead of fearing the upgrade.
Keep the loop closed
Production traces feed new cases; new cases tighten the suite; the suite gates releases. Teams that run this loop improve steadily. Teams that do not tend to oscillate between changes they cannot evaluate and fears they cannot quantify.