Understanding Context Windows in LLMs

A large context window is useful. It lets us give a model more of a document, conversation or codebase at once. But fitting the information into the request and getting the model to use it well are different problems.

I noticed this while using thinking models in Cursor: as the context grew substantially, the model would sometimes say it had edited a file or taken an action without actually doing it. That was an observation from using the tool, not a controlled test of context length. Still, it is a good reason to check the work rather than assume that a large window solves the problem.

What goes into the window?

The context window is a limit on the tokens available to a model for a request. The exact accounting depends on the model and API. Instructions, conversation history, retrieved material and tool definitions can all contribute to the input. The answer needs room too, including reasoning tokens where the model counts those toward output.

So the budget starts with a simple relationship:

Input budget = context limit - output reserve - safety margin

The output reserve should match the output limit configured on the actual request. It also has to fit the model's separate maximum output limit. A safety margin is useful for uncertainty; it is not a substitute for counting the complete request.

Character counts are only rough estimates of token counts. Code, different languages and message formatting can behave quite differently. For a real request, use the relevant tokenizer or provider's token-counting endpoint, including the fields you will send.

Keeping a conversation within budget

Dropping the oldest messages sounds straightforward, but it can also drop the system instructions or separate a tool call from its result. A useful illustration should make those choices explicit.

The function below handles text-only conversations with one system message, completed user/assistant turns, and a new user message. It preserves the system message and the new question, then keeps the most recent complete turns that fit. It deliberately does not handle tool calls, images, files or developer messages.

def fit_text_history(system, history, question, *,
                     context_limit, output_reserve,
                     safety_margin, count_request):
    """history is a list of (user_text, assistant_text) pairs.

    count_request(messages) must count the full serialized input
    for the chosen model, including message-format overhead.
    """
    if context_limit <= 0 or output_reserve <= 0 or safety_margin < 0:
        raise ValueError("Invalid token budget")

    input_budget = context_limit - output_reserve - safety_margin
    if input_budget <= 0:
        raise ValueError("No room left for input")

    system_message = {"role": "system", "content": system}
    question_message = {"role": "user", "content": question}

    def request(turns):
        messages = [system_message]
        for user_text, assistant_text in turns:
            messages.extend([
                {"role": "user", "content": user_text},
                {"role": "assistant", "content": assistant_text},
            ])
        return messages + [question_message]

    if count_request(request([])) > input_budget:
        raise ValueError("System instructions and question do not fit")

    kept = []
    for turn in reversed(history):
        candidate = [turn] + kept
        if count_request(request(candidate)) > input_budget:
            break
        kept = candidate

    return request(kept)

Notice that it counts each candidate request as a whole. It does not assume that counting message text alone accounts for the input. If the fixed instructions and current question do not fit, it raises an error rather than silently discarding either one.

It also stops at the first turn that would exceed the budget. That keeps a contiguous recent portion of the conversation. Skipping a large recent turn to include smaller, older ones might fit more text, but it can leave a misleading conversation. Even complete turns may depend on earlier context, so this is a budgeting example, not a guarantee that the remaining history is sufficient to answer.

For conversations involving tools, preserve the tool-call and tool-result groups required by the API. If removing history would lose a decision the next step needs, summarize it deliberately or keep it as explicit state.

What fits is not always what helps

The Lost in the Middle study found that performance on its retrieval tasks could depend on where relevant information appeared in a long input. That is a finding about the tested models and tasks, not a rule that moving every important instruction to the end fixes every request.

A more useful question is: what does the model need for this particular step?

For an edit, that may be the relevant code, the constraint it must preserve and the test that should pass. For a question about a document, it may be a specific section plus enough surrounding material to understand it. Repeated failed attempts and unrelated conversation can consume space without making either task easier.

Summaries can help, but they are lossy. Keep exact values, decisions and source references outside a summary when losing them would change the answer. The same applies to passing information between agents: a short handoff is only useful if it preserves what the next agent actually needs.

Retrieval is a selection problem

Embeddings can help find semantically related material. That does not make arbitrary chunks the right unit of context for every task. The chunk boundaries, document structure, query and evaluation all matter.

In Thoughts on LLMs, I was critical of using vector embeddings over arbitrary chunks as a default RAG design. The distinction is between a useful retrieval technique and assuming that one technique is the whole system. Sometimes a section, an exact search or a filesystem lookup gives the model a better starting point. Test against the questions the system needs to answer.

Adjusting context as the task changes

SeamlessGen is one way I explored this idea: pause at a generation boundary, adjust the instructions, then resume with the output so far. It makes the relevant phase explicit instead of relying only on the size of the window.

There is no single context strategy that settles every task. Budget the request, preserve the information that cannot be lost, and verify the result. A model saying it completed an action is still different from the action having happened.