A cheaper call needs several attempts before producing a usable answer. Did switching models save money? Count all the work required for a successful task, then compare quality and latency under the same workload.
Before you begin: You understand token usage, retrieval, caching, and task evaluation.
Use cost per successful task as a central measure, alongside quality and latency. Include failed attempts, tool usage, retrieval, storage, and infrastructure where they apply.
Make the accounting explicit
Hosted model pricing may distinguish input, output, cached input, and other usage categories. Reasoning-related usage and multimodal inputs can have additional rules. Use current provider documentation and the actual usage response; do not assume every token category has the same rate.
The following prices are fictional so the arithmetic stays useful without becoming an outdated price table.
# Runnable: Python 3, standard library.
from decimal import Decimal
input_tokens = Decimal(12000)
output_tokens = Decimal(2000)
input_per_million = Decimal('1.00') # Fictional price.
output_per_million = Decimal('4.00') # Fictional price.
cost = ((input_tokens * input_per_million
+ output_tokens * output_per_million) / Decimal(1000000))
assert cost == Decimal('0.020')
print('Illustrative request cost:', cost)
This is one request, not a complete monthly estimate. Multiply by the real workload and add the other relevant categories, including failed requests that incur charges.
Remove waste before weakening the model
Inspect repeated instructions, unused history, duplicate retrieved passages, oversized outputs, and retries that repeat the same failure. A shorter prompt is useful when it preserves necessary information. Removing the exception from a policy to save tokens is not an optimization.
Use output limits appropriate to the task. A classification response should not require a long essay. Structured output can reduce parsing ambiguity, but it does not eliminate the need to validate correctness.
Caching can save repeated work when its scope and freshness are correct. Batch processing may suit noninteractive tasks, while a phone conversation needs acceptable response latency. Match the method to the user journey.
Include the failed tasks in the bill
Use fictional costs: system A spends one unit on each of 100 attempts and completes 50 tasks. System B spends 1.5 units on each of 100 attempts and completes 90. A costs two units per successful task; B costs about 1.67. B's call is more expensive while its useful result is cheaper.
What could make this comparison unfair?
Different task mixes, success criteria, or omitted fallback costs. Compare the same intended workload and count all attempts, including failures. If A can retry, add the actual retry cost and resulting successes rather than assuming every retry succeeds independently. Also report latency and any unacceptable quality regressions.
Build this calculation from observed usage when available. Keep the fictional example separate from current provider prices. The arithmetic teaches a denominator; it does not estimate your application's traffic, savings, or retention.
Route based on measured difficulty
A smaller model may handle routine extraction, while a stronger model handles ambiguous synthesis or difficult tool planning. Build routing rules from labeled failures and compare the whole routed system with a single-model baseline.
Fallback has a cost. If the first model nearly always fails before escalation, the router adds latency and expense. Use signals that predict whether the cheaper path can meet the task standard, and evaluate misrouting in both directions.
Self-hosting shifts cost toward hardware, utilization, operations, and capacity risk. A low theoretical cost per token at full utilization may not describe a lightly used service with idle GPUs.
Watch outcomes together
Track total spend, successful tasks, quality failures, retries, cache hits, and latency by task type. Set budgets and alerts around observed usage. Do not claim savings until you compare representative before-and-after traffic or a controlled workload.
Exercise: prompt compression cuts input tokens by 40% but raises the retry rate substantially. What decides whether it helped?
Compare your reasoning
Total cost and latency per successful task, plus quality and regression checks. The token reduction is one component; repeated calls can erase it. Inspect whether essential evidence was removed.
Next, instrument the system so those decisions come from traceable measurements.
Sources
The vLLM documentation describes serving mechanisms relevant to self-hosted efficiency. Training Compute-Optimal Large Language Models concerns training allocation; keep that distinct from the application-serving cost model taught here.
Continue: Monitoring and Observability.