Evaluating & Testing AI Agents
Agent evaluation is harder than chatbot evaluation because agents take steps.
You must evaluate the final answer and the path used to get there.
What to measure
| Area | Questions |
|---|---|
| task success | did the user goal get completed? |
| tool correctness | were the right tools called with valid args? |
| safety | did it avoid forbidden or risky actions? |
| efficiency | did it waste steps, tokens, or time? |
| recovery | did it handle tool failures? |
| grounding | did claims cite reliable evidence? |
| human burden | how much correction was needed? |
Trace-based evaluation
An agent trace should show:
text
user goal
plan
tool call
tool result
state update
decision
final answer
Without traces, you can only judge the final output. That hides bad behavior.
Test cases
Include:
- easy successful task
- ambiguous task
- missing information
- bad tool result
- permission-denied tool result
- prompt injection in retrieved content
- high-risk action requiring approval
- task where the correct answer is to stop
Rubric example
| Score | Meaning |
|---|---|
| 5 | completed task safely with minimal steps |
| 4 | completed task with minor inefficiency |
| 3 | partially completed or needed human correction |
| 2 | wrong result but no harm |
| 1 | unsafe, unauthorized, or misleading behavior |
Automated plus human eval
Use automated checks for:
- schema validity
- tool argument validity
- max step count
- citations present
- policy violations
Use human review for:
- usefulness
- judgment quality
- business risk
- unclear edge cases
Regression workflow
text
save traces -> change prompt/model/tool -> rerun tasks -> compare success, cost, safety
Do not ship an agent change because one demo looked good.
Knowledge check
Q1: Why are traces necessary for agent evaluation?
They show whether the agent used tools, evidence, and decisions correctly before the final answer.
Q2: What is a good high-risk test case?
A case where the agent must pause and ask for approval instead of acting alone.