Cost Optimization
LLM cost is a product design problem, not just an engineering problem.
You reduce cost by sending less work to expensive models while preserving quality.
Cost drivers
| Driver | How to reduce it |
|---|---|
| input tokens | shorten context, retrieve better, compress history |
| output tokens | set max output and clearer formats |
| reasoning tokens | route hard tasks only |
| model choice | use smallest model that passes evals |
| repeated prefixes | prompt caching |
| repeated queries | response or retrieval caching |
| retries | better validation and error handling |
Model routing
Route by task:
text
simple extraction -> fast model
RAG answer -> balanced model
hard debugging -> reasoning model
large document -> long-context model
high-risk action -> model + human review
Token budget
Every feature should have a budget:
- max input length
- max retrieved chunks
- max output tokens
- max reasoning effort
- max tool calls
- max retries
Budgets make cost predictable.
Do not optimize blindly
Cheap but wrong is expensive.
Use evals to compare:
- quality
- latency
- total cost
- human corrections
- failure rate
Practical wins
- summarize old chat history
- retrieve top evidence instead of full docs
- use structured outputs to avoid long prose
- cache stable prefixes
- batch offline jobs
- move routine tasks to smaller models
Knowledge check
Q1: What is the safest model-routing rule?
Use the cheapest model that passes task-specific evals.
Q2: Why can reducing output length save more than expected?
Output tokens are often slower and more expensive than input tokens.