Would you release an agent that answers accurately but sometimes reads another user's record? To evaluate an agent, you need to inspect the answer, the actions, and the final state together.
Before you begin: Understand tool-using loops, held-out tests, and task state.
The first violated the task boundary, despite its correct text. The second may have handled failure properly. Agent evaluation needs to inspect outcome, evidence, actions, and limits together.
Define a successful task before running it
For a workshop booking assistant, success might mean creating exactly one requested booking for the authenticated user and accurately reporting its status. If the workshop is full, success means reporting that outcome without inventing a confirmation.
Keep the expected final state with the test case. A text-only grader can miss duplicate bookings, wrong owners, or an action the user never requested. Inspect the backing records when the task changes state.
Use a rubric for criteria that need judgment: whether the answer addresses the question, explains uncertainty, and cites support. Separate these from deterministic checks such as valid IDs or exactly one write.
Create a small but varied fixture set
Include an ordinary task, an ambiguous request, missing evidence, a denied operation, a temporary tool failure, and an uncertain result after a possible commit. Add a retrieved document that contains irrelevant instructions.
Use deterministic mock tools for control-flow tests. They make failure timing reproducible without touching real accounts. Then run integration tests against a safe test environment to cover the actual API boundary. Mocks alone do not prove the service is wired correctly.
Record model revision, prompts, tools, generation settings, budgets, and fixture versions. Without those, a changing score is hard to explain.
Measure repeatability, not a single lucky run
A stochastic agent may succeed once and fail on the same task later. Repeated trials help distinguish occasional success from consistent behavior. The following Python 3 example uses fictional observations.
# Runnable: Python 3, standard library.
trials = {
"lookup": [True, True, True],
"booking": [True, False, True],
"missing-time": [False, False, True],
}
all_results = [result for values in trials.values() for result in values]
per_trial = sum(all_results) / len(all_results)
always_passed = sum(all(values) for values in trials.values())
print("Observed trial success:", round(per_trial, 3))
print("Tasks passing every trial:", always_passed, "of", len(trials))
assert always_passed == 1
assert sum(all_results) == 6
The overall trial success is two-thirds, but only one task passes all three trials. This tiny sample does not establish a population reliability estimate. It demonstrates why a single successful screenshot is weak evidence.
“At least one success across several attempts” is a different metric from “every attempt succeeds.” If a production user gets one attempt, an evaluation that chooses the best of many must disclose that mismatch.
Keep the hard failures visible
Consider two fictional candidate runs. Candidate A answers eight of ten tasks correctly, and every action stays within its permissions. Candidate B answers nine correctly, but one run also reads a private record belonging to another user. A single answer-accuracy score makes B look better. Does that match your release requirement?
Write two separate results for each run: whether the task outcome passed and whether the authorization boundary held. Add a timeout case in which the agent truthfully reports that it could not finish. Decide beforehand whether that is an acceptable handled failure or an unsuccessful task. Keep both labels if they answer different questions.
Choose the release evidence
If access isolation is a required condition, B fails that condition. You can still report its answer score, but should not average the privacy violation into an otherwise favorable rating. A handled timeout can pass the recovery check while failing task completion. Distinct labels preserve that information and tell the team what to fix.
Neither ten tasks nor a clean result proves universal safety. Keep the tested scope visible and add cases when an actual failure reveals a missing boundary.
Diagnose the first meaningful failure
Inspect the trajectory: did retrieval return the needed fact? Were tool arguments valid? Did the server authorize the operation? Did the agent interpret the result correctly? Did it stop within its budget?
The last visible mistake may have begun earlier. A writer cannot reliably recover an exception removed during context compression. A booking summary cannot establish success when the tool result remained unknown.
Use model judges cautiously. Calibrate them against human-reviewed examples, test order and verbosity bias, and preserve disagreements for inspection. Another model's approval does not certify an external state change.
Practice a release decision
The new agent improves answer ratings but starts making more unnecessary calls and sometimes exceeds its deadline. Should it replace the baseline?
Compare your decision
Compare the complete task criteria, including timing and usage limits. Inspect which cases improved and which regressed. A better prose rating does not automatically outweigh a broken deadline. Revise or limit the change until the intended user journey meets its requirements.
Maintain a compact regression set from real failures after removing unnecessary private data. Keep a separate final evaluation set so repeated tuning does not turn the entire benchmark into training material.
WebArena evaluates tasks in realistic web environments. HELM illustrates the need for multiple evaluation dimensions. The fictional trial counts here are original examples, not reported benchmark results.
Continue: Production agent project, where you can apply these checks to a recoverable pipeline.