If three frameworks all produce a plausible answer, how will you choose between them? Build the same small evidence task in each, then compare what happens when the evidence is missing or misleading.
Before you begin: Complete the ADK, OpenAI Agents SDK, and CrewAI lessons. You may do the design and fixture evaluation without paid API calls.
Fix the task before changing the framework
Your assistant reads a fictional workshop catalog and explains whether the records answer a user's question. It may read records; it may not book, email, or modify anything. The final result must contain an answer, source IDs, and a status such as supported, missing, or conflicting.
Keep the catalog, questions, evaluation rules, and model family as comparable as practical. If a provider integration requires a different model or configuration, record that difference. You are comparing whole implementations under those conditions, not isolating a pure framework effect.
Use this tiny catalog as a starting fixture:
| Source | Record |
|---|---|
| N1 | W1 capacity: 12; starts at 15:00; date: October 3 |
| N2 | W2 capacity: 8; starts at 10:00; date: October 4 |
| N3 | W1 booking status: not supplied |
First answer “Can 10 people fit in W1?” by hand. Capacity supports yes. It does not support availability or a confirmed booking. Write those excluded claims into your evaluation rule.
Can the same failure be recovered in every framework?
Separate the task from the orchestration library. Keep the model, tools, input cases and acceptance rules fixed while comparing implementations. Otherwise you may attribute a better result to the framework when a different prompt or model caused it.
Include an interrupted run and a tool failure alongside the happy path. Inspect the state you can recover and the effort required to understand a failure. The challenge evaluates a framework as part of an operating system for your task, not as the most impressive demonstration in a gallery.
Each framework runs its own showcase with different models and tools.
Implement one shared task and inject the same failure in each version.
Differences in recovery, clarity and operating effort become comparable.
Make a common boundary
Define each implementation as an adapter: it receives a question and the permitted lookup, then returns a result in your common format. ADK and the OpenAI runner can use a single tool-using agent. CrewAI can use extraction followed by writing. Do not add extra agents merely to make the diagram symmetrical.
The adapter contract below is illustrative JSON, not observed model output:
{
"answer": "W1 has capacity for 10 people; availability is unknown.",
"source_ids": ["N1", "N3"],
"status": "supported"
}
Parse the result and verify that cited IDs exist. Then examine whether those sources actually support the claim. JSON validity and citation existence are useful checks, but neither establishes entailment: whether evidence really implies the answer.
Build the evaluation before running the experiment
Use at least these six cases: a known capacity, an unknown workshop, a request to book, a question about remaining seats, conflicting dated records, and a tool timeout. Each case needs an acceptable outcome and a failure example. Do not score a timeout as “no workshops found.”
For example, a successful unknown-workshop response states that the record is missing and does not invent a capacity. A failed response confidently substitutes W1. This rule is more useful than checking for the word “sorry.”
Run each case more than once if sampling varies. Save model identifiers, dependency versions, instructions, tool events, outputs, latency, and usage when available. Redact credentials and private input. A small table of observed cases is enough; leave cells blank when you did not run an integration.
Investigate a tempting winner
Implementation A answers all known-record cases quickly but invents an answer during a timeout. B is slower and reports the missing evidence. Which should win?
Compare the decision
For a product that promises source-backed answers, A fails a release requirement. Evaluate correctness before optimizing latency among acceptable implementations. B is not universally superior; it is preferable under this contract and this evidence. Improve A's error path, rerun the same cases, and keep the original results for comparison.
Finish with a defensible choice
Deliver the fixture, adapter code, case table, actual results, and a short decision note. Explain the failure you learned most from, one tradeoff you accepted, and what would cause you to reconsider. If you only completed the fixture and design, label the project as a design exercise rather than a tested framework benchmark.
Your next integration problem is connecting these agents to services without rewriting every connection for every host. That is the role MCP aims to address.
Practice with feedback
Compare frameworks on one fixed task
Three frameworks are candidates. Each has a demo that looks impressive on its own example.
Hold the task and evaluation constant so the comparison is about the framework.
Check your understanding
Your task
Write the comparison protocol and the decision, including what you would give up.
These notes stay on this page. Download them before leaving.
What to include
- Tools are shared, with adapters named
- The fixture includes cases designed to fail
- Trace quality is assessed concretely, for example by debugging one induced failure
- The exit cost is stated
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Fixed task: given a member email, identify the class and report places left, or say it cannot be determined. Shared tools: find_class and places_left, implemented once in tools.py; each framework gets a 10-line adapter. Fixture inputs: 40 emails. 25 straightforward, 10 ambiguous (two possible classes, vague descriptions), 5 impossible (classes we do not run). The 15 hard ones are where the comparison lives. Evaluation: success is the correct class and correct number, or a correct 'cannot determine' on the 5 impossible ones. Measured by a script against hand labels; no model judges.
| Framework | success | avg tool calls | p95 latency | trace quality | deps | |---|---|---|---|---|---| | A | 34/40 | 2.4 | 3.1s | full call args + raw results, JSON | 4 | | B | 36/40 | 4.9 | 7.8s | prose log, args truncated | 31 | | C | 33/40 | 2.1 | 2.6s | full, plus replay from trace | 6 |
The tempting winner and why I resisted: B has the best success count. It gets there with twice the tool calls, and its 2 extra wins are both ambiguous emails where it guessed and happened to be right; on the 5 impossible emails it answered confidently 3 times. Its trace truncates arguments, so when I induced a failure I could not tell which slug it had queried. A higher score reached by guessing is not a better system for this task.
Decision: A. Comparable accuracy, honest behaviour on impossible inputs, traces I could actually debug, and 4 dependencies rather than 31. What I would need to leave it: our tool logic is framework-independent and the adapter is 10 lines, so switching is under a day. I would move if A's trace format regressed or if we needed multi-agent handoffs, which A does not model well.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Sources
ADK evaluation introduces evaluation of agent behavior. OpenAI Agents SDK and CrewAI documentation are the implementation references, not independent comparative benchmarks.