Long context: test whether the model uses what it can receive

A handbook fits into the request, but the model misses a crucial exception. What did the large context window guarantee? It allowed the input under certain limits; evidence use and correct synthesis still need their own tests.

Before you begin: Understand tokens, retrieval, attention, and the key/value cache.

A context window is a capacity limit on what a model can process in a request under a particular interface. It is not a guarantee that every detail receives equal attention or that a long answer will use the right evidence. Some interfaces budget input, output, and reasoning differently, so check the exact model's documented limits.

Separate three questions

First, can the request be accepted without truncation? Second, can the model find the relevant evidence? Third, can it combine that evidence into a correct answer?

A simple “needle in a haystack” test answers part of the second question. It does not establish that a model can resolve conflicting policies, join several records, or preserve a condition across a long document. Passing one synthetic lookup is useful evidence with a narrow scope.

Longer input also increases prefill work and cache memory. Some architectures or attention patterns alter those costs, but a large advertised limit alone does not specify performance.

Build a position-sensitive example

Use a fictional workshop policy with a general rule and an exception:

• General rule: cancellations require seven days' notice.
• Exception: workshops canceled by the organizer receive a full refund.
• Case: the organizer canceled Maya's workshop yesterday.

The correct answer applies the exception. Now place that exception at the beginning, middle, and end while keeping the wording and question fixed. Add plausible distractors, including an older policy clearly labeled as obsolete.

Record whether each answer names the correct condition and cites the supporting passage. This changes evidence position while preserving the answer, making the result easier to interpret than an unrelated collection of prompts.

# Runnable: Python 3, standard library.
# This builds text fixtures; it does not call or evaluate a model.
rule = "Cancellations normally require seven days' notice."
exception = "Organizer cancellations receive a full refund."
filler = ["This section describes room equipment."] * 30

fixtures = {}
for name, position in [("start", 0), ("middle", 15), ("end", 30)]:
    sections = filler.copy()
    sections.insert(position, exception)
    fixtures[name] = rule + "\n" + "\n".join(sections)

assert all(text.count(exception) == 1 for text in fixtures.values())
print(list(fixtures))

These fixtures are short and deliberately artificial. For a real evaluation, add long documents from the intended workload, use the exact tokenizer to measure length, and ensure filler does not introduce accidental clues.

Move two facts independently

The fixture tests one exception at different positions. Extend it with a second fact identifying who canceled the event. Put the policy exception near the beginning and the cancellation record near the end, then swap them while keeping the answer unchanged.

Keep lengths and wording as controlled as practical, and record the tokenizer count. The short fixture program only constructs inputs; it does not evaluate a model. Use realistic documents as well before deciding that a product supports long-document comparison.

Decide what deserves context

More text can help when the task needs broad coverage, such as comparing several policies. It can hurt when irrelevant or conflicting passages distract the model. Remove duplicate navigation, preserve document boundaries, and identify dates and source authority.

Retrieval and long context can work together. Retrieve relevant sections, then include enough surrounding material to preserve qualifications. Alternatively, use a whole-document baseline when the document is manageable. Compare both on the same questions and evidence, including latency and cost.

Do not compress away the only sentence that changes the answer. A summary is another transformation that needs evaluation.

Check the request actually sent

The UI may display a full document while the backend truncates it. Inspect token budgeting, message serialization, attachment extraction, and error handling. Reserve room for the expected answer according to the interface's documented behavior.

For multi-turn conversations, an application may summarize or discard older turns. Tell the learner which sources remain available. A large model window cannot restore evidence that the application never sent.

Practice an evaluation decision

A model passes every single-fact lookup but fails questions requiring two distant passages. Is the long-context feature ready for document comparison?

Lost in the Middle studies position-sensitive evidence use. Its historical results motivate these tests; they do not assign a fixed failure rate to current models.

Continue: Multimodal models, where the application must preserve evidence from images and audio as carefully as text.