Would you release an agent that answers accurately but sometimes reads another user's record? To evaluate an agent, you need to inspect the answer, the actions, and the final state together.
Before you begin: Understand tool-using loops, held-out tests, and task state.
The first violated the task boundary, despite its correct text. The second may have handled failure properly. Agent evaluation needs to inspect outcome, evidence, actions, and limits together.
Define a successful task before running it
For a workshop booking assistant, success might mean creating exactly one requested booking for the authenticated user and accurately reporting its status. If the workshop is full, success means reporting that outcome without inventing a confirmation.
Keep the expected final state with the test case. A text-only grader can miss duplicate bookings, wrong owners, or an action the user never requested. Inspect the backing records when the task changes state.
Use a rubric for criteria that need judgment: whether the answer addresses the question, explains uncertainty, and cites support. Separate these from deterministic checks such as valid IDs or exactly one write.
Create a small but varied fixture set
Include an ordinary task, an ambiguous request, missing evidence, a denied operation, a temporary tool failure, and an uncertain result after a possible commit. Add a retrieved document that contains irrelevant instructions.
Use deterministic mock tools for control-flow tests. They make failure timing reproducible without touching real accounts. Then run integration tests against a safe test environment to cover the actual API boundary. Mocks alone do not prove the service is wired correctly.
Record model revision, prompts, tools, generation settings, budgets, and fixture versions. Without those, a changing score is hard to explain.
Was that success repeatable?
One run can be unusually fortunate. Repeat fixture tasks under controlled conditions and record both success and the cost of attempts. A system that succeeds intermittently may still be useful, but its reliability is different from the result of one selected demonstration.
Reset relevant state between independent trials and preserve traces of failures. Evaluate final effects, evidence and forbidden actions as well as the text response. The challenge asks you to describe variability honestly and identify which failures need deterministic safeguards around the agent.
The same ten tasks yield 9, then 6, then 8 successes across three runs.
Report all runs and inspect recurring failure categories instead of selecting the best run.
The evaluation exposes variability and gives a more useful basis for release decisions.
Measure repeatability, not a single lucky run
A stochastic agent may succeed once and fail on the same task later. Repeated trials help distinguish occasional success from consistent behavior. The following Python 3 example uses fictional observations.
# Runnable: Python 3, standard library.
trials = {
"lookup": [True, True, True],
"booking": [True, False, True],
"missing-time": [False, False, True],
}
all_results = [result for values in trials.values() for result in values]
per_trial = sum(all_results) / len(all_results)
always_passed = sum(all(values) for values in trials.values())
print("Observed trial success:", round(per_trial, 3))
print("Tasks passing every trial:", always_passed, "of", len(trials))
assert always_passed == 1
assert sum(all_results) == 6
The overall trial success is two-thirds, but only one task passes all three trials. This tiny sample does not establish a population reliability estimate. It demonstrates why a single successful screenshot is weak evidence.
“At least one success across several attempts” is a different metric from “every attempt succeeds.” If a production user gets one attempt, an evaluation that chooses the best of many must disclose that mismatch.
Keep the hard failures visible
Consider two fictional candidate runs. Candidate A answers eight of ten tasks correctly, and every action stays within its permissions. Candidate B answers nine correctly, but one run also reads a private record belonging to another user. A single answer-accuracy score makes B look better. Does that match your release requirement?
Write two separate results for each run: whether the task outcome passed and whether the authorization boundary held. Add a timeout case in which the agent truthfully reports that it could not finish. Decide beforehand whether that is an acceptable handled failure or an unsuccessful task. Keep both labels if they answer different questions.
Choose the release evidence
If access isolation is a required condition, B fails that condition. You can still report its answer score, but should not average the privacy violation into an otherwise favorable rating. A handled timeout can pass the recovery check while failing task completion. Distinct labels preserve that information and tell the team what to fix.
Neither ten tasks nor a clean result proves universal safety. Keep the tested scope visible and add cases when an actual failure reveals a missing boundary.
Diagnose the first meaningful failure
Inspect the trajectory: did retrieval return the needed fact? Were tool arguments valid? Did the server authorize the operation? Did the agent interpret the result correctly? Did it stop within its budget?
The last visible mistake may have begun earlier. A writer cannot reliably recover an exception removed during context compression. A booking summary cannot establish success when the tool result remained unknown.
Use model judges cautiously. Calibrate them against human-reviewed examples, test order and verbosity bias, and preserve disagreements for inspection. Another model's approval does not certify an external state change.
Practice a release decision
The new agent improves answer ratings but starts making more unnecessary calls and sometimes exceeds its deadline. Should it replace the baseline?
Compare your decision
Compare the complete task criteria, including timing and usage limits. Inspect which cases improved and which regressed. A better prose rating does not automatically outweigh a broken deadline. Revise or limit the change until the intended user journey meets its requirements.
Maintain a compact regression set from real failures after removing unnecessary private data. Keep a separate final evaluation set so repeated tuning does not turn the entire benchmark into training material.
WebArena evaluates tasks in realistic web environments. HELM illustrates the need for multiple evaluation dimensions. The fictional trial counts here are original examples, not reported benchmark results.
Practice with feedback
Measure repeatability, not a lucky run
An agent completes 9 of 10 fixture tasks. Rerunning the same fixtures gives 6, then 8.
Check outcome and route, with a varied fixture set and repeated runs.
Check your understanding
Your task
Design the evaluation and the release rule for a stochastic agent.
These notes stay on this page. Download them before leaving.
What to include
- Four fixture categories with counts
- Route checks are defined concretely
- Repeats are stated and the spread is reported
- The release rule uses a lower bound, not the best run
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Fixture set, 44 tasks: typical: 20 (ordinary member questions with a clear answer) should-fail: 8 (classes that do not exist, dates in the past) refusal-required: 10 (requests for another member's details, or for actions outside the agent's permissions) tool-failure-injected: 6 (timetable service returns 503, or an empty list, or malformed JSON)
Per task: outcome check: a script comparing the final answer against the expected value, or asserting the refusal string, or asserting an honest 'unavailable' for the injected-failure tasks. route check: assertions over the trace - no write tools called on read-only tasks, no more than 6 tool calls, no repeated identical call, and every claim's source id present in the trace. Runs per task: 5, at the production temperature. I report median and the min-max range, never a single figure.
Results, with spread: typical 19/20 (range 18-20), should-fail 8/8 (8-8), refusal 9/10 (8-10), injected-failure 3/6 (2-4). Overall median 39/44 with a range of 36-42. Release rule: I ship on the worst observed run, not the median. Requirement: minimum across 5 runs at or above 36/44, refusal category never below 8/10, and zero write-tool calls on read-only tasks in any run. The last one is a hard gate with no tolerance, because it is a permissions violation rather than a quality shortfall. The first meaningful failure and what it told me: the injected-failure category at 3/6. On a 503 the agent retried twice and then answered from its own prior, saying the class 'usually runs at 18:30'. The outcome check caught it; the route check explained it. That single category is now the one I look at first, because it is the one where the agent stops being grounded.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
, where you can apply these checks to a recoverable pipeline.