An answer was slow and wrong, but the log says “success.” Which success did it record? Observability should connect transport, model, tool, and task outcomes without collapsing them into one status.
Before you begin: You understand requests, retrieval stages, tool calls, and basic latency statistics.
Observability combines metrics for patterns, traces for individual request paths, and logs for relevant events. The goal is to explain behavior, not to collect every possible byte.
Trace one task across its stages
Give the task a correlation identifier and connect spans for authentication, retrieval, reranking, generation, tool execution, and final validation. Record versions and structured outcomes so a regression can be associated with a prompt, model, index, or deployment change.
For parallel work, parent duration is not the sum of every child duration. Two searches can overlap. Inspect the critical path and waiting time instead of adding all span durations as if they ran sequentially.
| Signal | Useful context |
|---|---|
| Model call | Model revision, usage, duration, and status |
| Retrieval | Index version, result IDs, and timing |
| Tool action | Operation ID, validated action type, and outcome |
| Final answer | Validation result and source references |
| User journey | Start, completion, cancellation, and recoverable failure |
Prefer identifiers and structured metadata over raw private documents. If prompt or response capture is necessary for debugging, apply deliberate access, retention, sampling, and redaction controls. Observability should not become an uncontrolled copy of user data.
Which span lies on the path the user waits for?
When stages overlap, adding their durations can exceed the end-to-end time. A trace shows when each span starts and finishes and which later work depends on it. The critical path is the chain of dependencies that determines when the result can be ready.
Draw the overlap before choosing an optimization. Speeding up work outside the critical path may reduce resource use without changing visible latency. The challenge asks you to use timing relationships and errors together rather than treating a dashboard of separate averages as an explanation of one slow request.
Retrieval and other preparation overlap during a request.
Add their durations as though they ran one after another.
The sum misrepresents end-to-end waiting; start times and dependencies are needed.
Averages hide the slow experience
# Runnable: Python 3, standard library.
import math
latencies_ms = [100, 110, 115, 120, 125, 130, 140, 150, 160, 2000]
ordered = sorted(latencies_ms)
def nearest_rank(values, percentile):
return values[max(0, math.ceil(percentile * len(values)) - 1)]
print('Median:', (ordered[4] + ordered[5]) / 2)
print('p95:', nearest_rank(ordered, 0.95))
assert nearest_rank(ordered, 0.95) == 2000
This uses the nearest-rank percentile definition on a tiny teaching sample. Monitoring systems can use other estimators. The lesson is that one slow request can matter even when most requests are quick; report sample counts and enough traffic for meaningful trends.
Find the critical path in overlapping work
Two independent searches start together and take 400 ms and 700 ms. Generation then takes 800 ms. Ignoring other overhead, the combined path is about 1,500 ms, not the 1,900 ms sum of both search durations and generation. The longer search controls when generation can begin.
Which search should you optimize first?
Under this dependency, reducing the 700 ms search can shorten the path until another dependency becomes limiting. Reducing the 400 ms search alone may not change completion time. Inspect actual start and end times, queueing, and dependencies before drawing this conclusion from production traces.
Now add a retrieval timeout that returns an empty list incorrectly. The latency may improve while answer quality worsens. Record semantic outcomes alongside durations so a faster failed stage cannot masquerade as a performance improvement.
Separate technical success from task success
HTTP 200 means the request protocol completed, not that the answer was correct. Track unsupported claims, invalid structured outputs, failed tool outcomes, and user-reported problems separately from transport errors.
An agent can consume many steps without making progress. Record repeated actions, budget exhaustion, and stop reasons. A task that ends in an honest partial result should be distinguishable from one that falsely claims completion.
Use alerts for actionable conditions, such as a sharp error increase after deployment or a broken core journey. Avoid alerting on every individual imperfect answer without a triage policy.
Exercise: model latency is unchanged, but end-to-end latency doubles. What should you inspect first?
Compare your reasoning
Queueing, retrieval, reranking, tool calls, network waits, and retries in the trace. Optimizing the model alone may not affect the new bottleneck.
Practice with feedback
Find the critical path in overlapping work
A request takes 4.2s. Retrieval takes 1.1s, reranking 0.9s, and generation 3.4s, and some of it runs concurrently.
Trace one task across stages and separate technical from task success.
Make the decision before reading the feedback
Check your understanding
Now make something you can check
Define the span structure, the critical path analysis, and the two success counters.
These notes stay on this page. Download them before leaving.
Check your reasoning against these points
- Spans have parents and attributes, not just names
- The critical path is derived from timestamps
- Task success has an automatic determination method
- One alert is deliberately rejected with a reason
Compare with a worked answer
Compare the decisions and the evidence. Your wording can be different.
Spans I emit: request (root; attributes: member scope, question length, request class), retrieve.lexical and retrieve.vector (siblings, run concurrently; attributes: k, filter, hits), rerank (attributes: candidates, model), generate (attributes: model, prompt sha, input tokens, output tokens, finish reason), validate (attributes: schema ok, citations resolved). Every span carries the trace id, which is also returned to the client and logged with the answer, so a complaint becomes a lookup.
How I identify the critical path: walk the span tree from the root and, at each level, follow the child whose end time is latest, since that is what the parent waited for. Durations alone cannot do this. The 4.2s request: the two retrieval spans run concurrently, 1.1s and 0.4s, so retrieval contributes 1.1s, not 1.5s. Rerank starts after both finish. But generation's span starts 0.3s before rerank ends, because we begin streaming the system prompt early. The critical path is 1.1 + 0.9 - 0.3 + 3.4 = 5.1s of work compressed into 4.2s wall clock, and generation is 81% of it. Any work on retrieval speed is work on the wrong stage.
Metrics technical: error rate by status, p50/p95/p99 latency per request class, token usage, and timeouts. task success: answered-correctly rate, refusal rate on answerable questions, and citation-resolution failures. how task success is determined automatically: the validator already checks that citations resolve and that refusals use the exact fixed string. A refusal on a question whose retrieval found chunks above threshold is counted as a task failure without any human involvement. A weekly sample of 30 is read by a person to calibrate that proxy.
The alert I would set: task-failure rate over a 30-minute window above 10%, because it fires on the failure users feel and it has fired for real reasons twice. The one I would not: an alert on average latency. It is the metric most likely to be quiet during exactly the incident I care about.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
Next, centralize routing and provider policy without hiding those outcomes behind a gateway.
Sources
The OpenTelemetry documentation explains traces, metrics, and logs. Its generative AI conventions page now points to a separate maintained repository. Follow that current reference and pin the convention and instrumentation versions used by your application. The older page is a migration pointer, not the current specification.