Long-Context Models (128K to 1M+ Tokens)
Modern models can accept very large inputs, sometimes hundreds of thousands or more tokens. That does not mean every application should paste everything into the prompt.
Long context is a capability, not a strategy. You still need context selection, ordering, compression, evaluation, and cost controls.
Why long context is useful
| Use case | Why it helps |
|---|---|
| codebase review | inspect many files together |
| legal or policy review | compare long documents |
| research synthesis | keep multiple sources in view |
| agent planning | preserve task history |
| multimodal files | handle large transcripts or documents |
The hidden costs
Long prompts can create:
- higher input cost
- slower first-token latency
- larger KV cache memory
- harder citation tracking
- lost-in-the-middle failures
- privacy exposure from unnecessary context
Long context vs RAG
| Approach | Best for |
|---|---|
| Long context | small number of highly relevant long artifacts |
| RAG | large changing corpus with search/filter needs |
| Hybrid | retrieve first, then use long context for selected documents |
Ordering matters
Place context intentionally:
text
system/developer instructions
task
short summary of sources
most important evidence
supporting evidence
output format
Do not bury the key evidence in the middle of unrelated text.
Practical rules
- Count tokens before sending.
- Remove irrelevant sections.
- Summarize repeated history.
- Keep source IDs with excerpts.
- Prefer retrieval for large corpora.
- Cache repeated prefixes where supported.
- Run evals for long-document edge cases.
Knowledge check
Q1: Why can long context hurt quality? Irrelevant or poorly ordered information can distract the model and hide important evidence.
Q2: When should you prefer RAG over long context? When the knowledge base is large, changing, searchable, or mostly irrelevant to each query.