Project: compare agent frameworks on one task

If three frameworks all produce a plausible answer, how will you choose between them? Build the same small evidence task in each, then compare what happens when the evidence is missing or misleading.

Before you begin: Complete the ADK, OpenAI Agents SDK, and CrewAI lessons. You may do the design and fixture evaluation without paid API calls.

Fix the task before changing the framework

Your assistant reads a fictional workshop catalog and explains whether the records answer a user's question. It may read records; it may not book, email, or modify anything. The final result must contain an answer, source IDs, and a status such as supported, missing, or conflicting.

Keep the catalog, questions, evaluation rules, and model family as comparable as practical. If a provider integration requires a different model or configuration, record that difference. You are comparing whole implementations under those conditions, not isolating a pure framework effect.

Use this tiny catalog as a starting fixture:

First answer “Can 10 people fit in W1?” by hand. Capacity supports yes. It does not support availability or a confirmed booking. Write those excluded claims into your evaluation rule.

Make a common boundary

Define each implementation as an adapter: it receives a question and the permitted lookup, then returns a result in your common format. ADK and the OpenAI runner can use a single tool-using agent. CrewAI can use extraction followed by writing. Do not add extra agents merely to make the diagram symmetrical.

The adapter contract below is illustrative JSON, not observed model output:

{
  "answer": "W1 has capacity for 10 people; availability is unknown.",
  "source_ids": ["N1", "N3"],
  "status": "supported"
}

Parse the result and verify that cited IDs exist. Then examine whether those sources actually support the claim. JSON validity and citation existence are useful checks, but neither establishes entailment: whether evidence really implies the answer.

Build the evaluation before running the experiment

Use at least these six cases: a known capacity, an unknown workshop, a request to book, a question about remaining seats, conflicting dated records, and a tool timeout. Each case needs an acceptable outcome and a failure example. Do not score a timeout as “no workshops found.”

For example, a successful unknown-workshop response states that the record is missing and does not invent a capacity. A failed response confidently substitutes W1. This rule is more useful than checking for the word “sorry.”

Run each case more than once if sampling varies. Save model identifiers, dependency versions, instructions, tool events, outputs, latency, and usage when available. Redact credentials and private input. A small table of observed cases is enough; leave cells blank when you did not run an integration.

Investigate a tempting winner

Implementation A answers all known-record cases quickly but invents an answer during a timeout. B is slower and reports the missing evidence. Which should win?

Finish with a defensible choice

Deliver the fixture, adapter code, case table, actual results, and a short decision note. Explain the failure you learned most from, one tradeoff you accepted, and what would cause you to reconsider. If you only completed the fixture and design, label the project as a design exercise rather than a tested framework benchmark.

Your next integration problem is connecting these agents to services without rewriting every connection for every host. That is the role MCP aims to address.

Sources

ADK evaluation introduces evaluation of agent behavior. OpenAI Agents SDK and CrewAI documentation are the implementation references, not independent comparative benchmarks.

Continue to the next lesson.