Caching: reuse work without reusing the wrong answer

Two learners ask “When is my booking?” The text matches exactly, but the answers should not. A cache can reuse work only when the inputs and permissions that determine the result are equivalent.

Before you begin: You understand model inputs, retrieval, authentication, and source versions.

Caching is a correctness decision as well as a performance technique. First identify exactly which computation is being reused and which inputs determine its validity.

Different caches reuse different things

Provider prompt caching is not necessarily a reusable final answer, and its eligibility and pricing depend on the provider. Check current documentation before promising savings. An exact response cache needs a broader key than the visible user question.

Include the inputs that can change the result

A cache key may need the user or tenant scope, model and prompt versions, generation settings, source snapshot, and relevant conversation state. If the answer depends on current permissions, revalidate them or use an invalidation strategy that reflects changes.

# Runnable: Python 3, standard library.
import hashlib
import json

def cache_key(user_scope, question, source_version):
    payload = {
        'scope': user_scope,
        'question': question,
        'source_version': source_version,
        'prompt_version': 'workshop-v2',
    }
    encoded = json.dumps(payload, sort_keys=True).encode()
    return hashlib.sha256(encoded).hexdigest()

a = cache_key('user-a', 'When is my booking?', 'v3')
b = cache_key('user-b', 'When is my booking?', 'v3')
updated = cache_key('user-a', 'When is my booking?', 'v4')
assert len({a, b, updated}) == 3
print('User and source changes produce different keys')

The hash provides a compact key, not encryption or access control. Sensitive cache values still need appropriate storage controls. This example also omits several inputs a real system may require.

Change the hidden input

Use the local key function with the same question and source version, then change the user scope. The key changes. Now keep all three arguments fixed but change the system prompt or model. The current toy key includes a fixed prompt version and omits the model, so it would reuse an entry unless you extend the contract.

For practice, classify a cached embedding, a retrieved candidate list, and a final answer. Their validity inputs differ. The exercise starts with identifying what was computed, then derives the key and invalidation rule from that operation.

Freshness needs an invalidation rule

A time-to-live limits how long an entry survives, but an urgent policy update may need immediate invalidation. Tie answers to source versions and invalidate derived responses when those sources change or access is revoked.

Do not cache transient failures as permanent facts. “Lookup timed out” is different from “no workshop exists.” If negative results are cached, give them a deliberate policy and a way to refresh.

Semantic similarity is not answer equivalence

“Can members cancel?” and “Can guests cancel?” may be close in embedding space but require different answers. Exact dates, negation, account identity, and permissions can matter more than overall wording similarity.

Use semantic caching only for a workload where equivalence can be evaluated, and validate important constraints before reuse. For personalized or changing facts, a conservative exact or retrieval cache may be easier to make reliable.

Exercise: a user loses access to a private document, but an answer based on it remains cached. Is a one-hour TTL enough by itself?

Next, measure the total cost of a successful task, including cache misses, retries, and evaluation.

Sources

The vLLM automatic prefix caching documentation describes computation reuse for matching prefixes. The LangChain caching documentation describes application-level response caching; these mechanisms have different validity rules.

Continue: Cost Optimization.