Why does an assistant lose track of a long conversation?

You tell an assistant that the museum is closed on Mondays. After a long conversation about transport and tickets, it proposes a Monday visit. Did it forget? Several different failures can produce that result, and they call for different fixes.

Before you begin: Know that a language model generates tokens using the context supplied to it.

Text takes up a budget

A tokenizer divides text into units and maps them to IDs. A token is not necessarily a word. Spaces, punctuation, common word fragments, and different writing systems can be represented differently. Two models can tokenize the same sentence into different numbers of tokens.

The context window limits how much tokenized information a model can handle in a request. Depending on the model and API, the budget may include input, generated output, and other token categories. Use the documentation for that particular model. A rough English words-to-tokens shortcut is not a reliable estimate for every language, code file, or image input.

The application also chooses what to send. It may keep recent messages, summarize older ones, retrieve selected notes, or reject an oversized request. A chat interface can display more history than the model receives on a particular turn.

Work through a fictional budget

Suppose a system allows 8,000 tokens total in a simplified input-plus-output budget. Instructions use 500, your question uses 300, and you reserve 1,200 for an answer. That leaves 6,000 for history and supporting material. If those materials use 7,000, something must change: shorten the input, reduce the planned output, select evidence, or use a compatible larger window.

These are invented limits for arithmetic practice. They are not the specification of a product. Real request formatting, tool schemas, or other inputs can consume space too, so a production application needs the provider's counting rules rather than this four-line estimate.

Available is different from used

If the Monday closure never reached the model, ask how the application selected context. If the fact was included but buried among irrelevant material, test how reliably the model uses it. If it saw two conflicting schedules, investigate which date and source it trusted.

A larger window may solve a capacity problem. It does not automatically solve evidence selection or contradictions. Research has found that the position of relevant information can affect performance in some long-context settings. That is a reason to test your own documents, not to assume every model fails in exactly the same position.

Build a small context experiment

Write a short fictional travel note with three constraints: the museum closes on Monday, the group arrives at noon, and the final bus leaves at 17:00. Ask for a visit plan using only those notes. Check each constraint.

Now add several paragraphs about unrelated attractions. Keep the question unchanged. Finally, move the three constraints from the start to the middle. Compare which facts survive in the plan. Do several trials if you use a model with variable outputs. Record the tool and date; the result belongs to that setup.

The goal is not to trick the assistant once. It is to distinguish a repeatable failure from a single unusual answer. If the plan fails, try a focused context containing only the relevant facts and see whether that resolves the problem.

Preserve what matters

For long work, keep a short, checked brief of the goal, constraints, decisions, and unresolved questions. Supply it when starting a new conversation. Do not compress away exceptions such as “open daily except Monday.” A summary that loses the exception may be shorter but less useful.

Treat summaries as derived documents that can contain mistakes. Keep the original source available for important details. When a system retrieves evidence, ask it to connect its claims to the actual passages instead of relying on a vague statement that it remembers.

Decide what went wrong

Your full transcript contains the closure fact, but the logged API request contains only the latest five messages. Should you fix the model's prompt, retrain the model, or change the application's context selection?

Next, we will write a prompt whose task and evidence are clear enough to evaluate.

Further reading

OpenAI's token explanation describes token counting for its products. Lost in the Middle studies information use across positions in long inputs.

Continue to the next lesson.