Context compression: remove repetition, preserve the conditions

A shorter evidence pack removes the exception that controls the answer. What did compression optimize? Before reducing tokens, identify the facts and qualifications the question requires you to preserve.

Before you begin: You understand retrieval and the need for source-grounded answers.

Compression is not automatically harmless. Dropping “except guest events” from a cancellation policy changes the rule. A shorter prompt can produce a faster wrong answer.

Choose what kind of reduction you need

First remove duplicates and irrelevant passages. This selection step often saves space without rewriting facts. Next, extract the specific sentences or structured fields that support the question, retaining their source locations.

Abstractive summarization rewrites information in new words. It can combine scattered material, but it can also omit a condition, merge conflicting sources, or introduce a claim. Treat a generated summary as a derived artifact that needs support, not as a replacement source of truth.

Make preservation requirements explicit

For the workshop policy, retain the rule, its scope, exceptions, dates, and responsible source. A query about refunds needs different context from a query about room access. Compression should be conditioned on the question while keeping enough evidence to detect misleading simplifications.

# Runnable: Python 3, standard library.
record = {
    'source_id': 'policy-v3',
    'rule': 'Cancel at least 24 hours before the session.',
    'exception': 'Guest events use their own cancellation terms.',
    'navigation': 'Home | Workshops | Contact',
}
required = ['source_id', 'rule', 'exception']
compressed = {key: record[key] for key in required}
assert compressed['exception'] == record['exception']
assert 'navigation' not in compressed
print(compressed)

This simple extraction makes the preserved fields inspectable. It does not solve arbitrary document understanding. For free text, use labeled examples and review whether qualifiers survive.

Compare two equally short summaries

Source: “Members cancel free until 24 hours before the session. Guest events follow their own terms.” Summary A says “Cancellation is free.” Summary B says “Members: free until 24 hours before; guest events: separate terms.” The second keeps the decision boundaries even though it remains compact.

For practice, create a preservation list for a table: row identity, column label, value, unit, date, and exception. Then compare extraction and generated summarization using the same budget. Report unsupported claims and omitted necessary facts separately; a summary can fail through either addition or deletion.

Budget with the actual model input

Count the system instructions, conversation, tool schemas, retrieved evidence, and reserved output together. A compressor that meets an evidence-only token target can still overflow the complete request.

Keep source IDs and permissions attached through every transformation. Do not combine passages with different access scopes into a shared summary that becomes visible to everyone. Summaries and caches need the same user or tenant boundaries as the source data they contain.

If compression removes a needed fact, fall back to a larger permitted context or report the limitation. Do not force a complete answer merely because the compressed form looks neat.

Check answerability before and after

Use questions about exceptions, numbers, units, negation, and conflicting versions. Compare whether a reviewer can answer from the original evidence and from the compressed evidence. Then compare generated answers and their citations.

Exercise: the original says “Free for members; guests pay 10 credits.” The summary says “The workshop is free.” Is this acceptable compression?

Next, connect ingestion, retrieval, compression, and answer checks into a complete production RAG project.

Sources

Lost in the Middle provides evidence that access to long context does not guarantee uniform use of it. Retrieval-Augmented Generation supplies the grounding context for distinguishing evidence from generated text.

Continue: Project: Production-Ready RAG System.