Prompt Caching and KV-Cache Strategies
Caching saves work you have already paid for.
LLM apps have several kinds of cache, and confusing them causes bad designs.
Cache types
| Cache | Stores | Best for |
|---|---|---|
| prompt/prefix cache | repeated prompt prefix computation | long system prompts and shared context |
| response cache | final answer | identical deterministic requests |
| semantic cache | answer for similar query | FAQ-style use cases |
| retrieval cache | search results | repeated queries over stable corpus |
| KV cache | attention state during generation | active inference requests |
Prompt caching
Prompt caching helps when many requests share the same prefix:
system prompt + policy docs + examples + user question
If the prefix repeats, providers or serving engines may reuse work.
Response caching
Only cache final responses when:
- the task is deterministic
- user permissions are identical
- data is not stale
- no private context leaks between users
Do not response-cache personalized or high-risk answers blindly.
Semantic caching
Semantic caching uses embeddings to find similar prior questions. It can reduce cost, but it can also return a wrong answer if two questions look similar but need different facts.
Use it only with strict freshness and confidence checks.
Practical checklist
- cache shared system prompts
- cache stable retrieval results
- include model and prompt version in cache keys
- include user/workspace permission scope
- set expiration times
- log cache hit rate
- bypass cache for high-risk tasks
Knowledge check
Q1: Why should prompt version be part of the cache key?
A new prompt can produce different behavior, so old cached answers may be invalid.
Q2: Why is semantic caching risky?
Similar wording does not always mean the same answer is correct.