Context window
Also: Context length
How much text a model can take into account at once, counted in tokens, and the multiplier a team applies to its own bill every time it fills one.
The context window is the span of tokens a model can attend to in a single request: the prompt, any retrieved material, the conversation so far and the answer being generated, all counted together. Published windows now reach a million tokens, which has moved the question from whether something fits to whether putting it there is worth paying for.
Why a larger window is not a free one
Every token in the window is a token billed and a token processed. A study of agent memory we reported showed what the choice costs: a curated set of retrieved guidelines added 5 percent to the tokens per run, from 110K to 116K, while the full guideline set added 51 percent, to 166K. On another model the full set pushed 148K to 263K, a 78 percent increase. The same memory configuration bought 16.1 points of task completion on one model and nothing at all on a third.
What it does not solve
Fitting more in is not the same as using it. Retrieval quality, ordering and the point at which a model stops attending all still apply, which is why context windows stopped being the constraint without ending the work of deciding what goes into them.
Questions
- What does context window mean?
- The number of tokens a model can take into account in one request, counting the prompt, any retrieved material, the conversation so far and the answer being produced. Published windows now reach a million tokens.
- Does a bigger context window cost more?
- Yes, in proportion to what goes into it. In one study of agent memory, a curated set of guidelines added 5 percent to tokens per run while the full set added 51 percent on the same model and 78 percent on another.