Design your evaluation around what can go wrong.
A small evaluation set becomes more useful when its failures are specific.
Name the failure before measuring it
“The assistant was wrong” is a starting observation, not a diagnosis. It may have used an outdated source, misunderstood the request, invented a detail, or called a tool with the wrong arguments. Those failures should not disappear inside one overall score.
Create a short failure taxonomy that matches the actual workflow. For a document assistant, this might include missing evidence, unsupported claims, wrong document version, and unnecessary refusal. For a tool-using agent, include unauthorized actions and incorrect parameters.
Include the uncomfortable examples
A useful test set contains ordinary questions as well as ambiguous requests, missing documents, contradictory sources, and out-of-scope tasks. Record where each example came from and who decided the expected behavior.
Keep sensitive data out of examples unless there is a clear, authorized reason to include it. Synthetic examples can help exercise a boundary, but label them and check that they represent a plausible situation.
Keep people in the review loop
Use explicit criteria that a reviewer can apply consistently. “Every policy claim is supported by the cited passage” is more actionable than “the answer seems good.” Review disagreements are useful evidence that the criteria or the task itself needs clarification.
Automated checks can validate structure, required fields, or tool arguments. More subjective judgments need calibration against human review. Do not assume that a second model’s score is a reliable measure simply because it is easy to generate.
Use evaluation to decide the next step
Review failures by severity and frequency. A rare action with serious consequences may deserve attention before a common wording issue. Decide whether the next change belongs in retrieval, permissions, prompting, tool design, or human review.
Save the system configuration alongside each run so a comparison can be reproduced. Expand scope only when the evidence supports it, and retain a clear path to stop or hand back a task when the system is outside its limits.