Monitoring and Observability
Traditional monitoring tells you whether a server is up. LLM observability tells you whether the AI system is behaving well.
What to trace
| Span | Useful fields |
|---|---|
| model call | model, prompt version, tokens, latency, cost |
| retriever | query, documents, scores, filters |
| tool call | tool name, arguments, result status |
| agent step | plan, action, observation, retry |
| evaluator | score, rubric, failure reason |
| user feedback | thumbs, correction, escalation |
Key metrics
- request count
- error rate
- p50/p95/p99 latency
- input, output, cached, and reasoning tokens
- cost per request
- schema validation failure rate
- no-answer rate
- groundedness score
- tool-call failure rate
- safety intervention rate
Why traces matter
When a user says "the answer was wrong," you need to know:
text
which prompt version ran?
which model answered?
which context was retrieved?
which tools were called?
did schema validation pass?
what did the evaluator score?
what changed since the last good run?
Production practices
- version prompts like code
- sample and redact logs
- connect traces to eval datasets
- alert on cost and quality regressions
- replay failed traces before changing prompts
- track model/provider changes
- keep retention rules for sensitive data
Agent observability
Agents need extra visibility:
- step count
- tool-call loops
- delegated agents
- approval points
- final state
- budget exhaustion
- unsafe action attempts
Knowledge check
Q1: Why are token metrics important? They determine cost, latency, context pressure, and sometimes reasoning effort.
Q2: What makes LLM observability different from normal API monitoring? You must monitor quality, grounding, safety, prompts, retrieved context, and tool behavior, not just uptime.