Introduction to context windows
A context window is the amount of conversation a language model can take into account when it writes a response. It covers the model's own messages as well as yours, plus any instructions and materials attached to the conversation. Its size is measured in tokens.
How big is a context window?
The size is set by the model itself. Early models were tiny by today's standards — GPT-3 could hold 2,048 tokens, three or four pages. Current models are much larger, typically holding between a couple of hundred thousand tokens and a million or more, which is thousands of pages.
Large does not mean unlimited. Anything past the window is simply not there for the model to read.
Context windows in a chat
In a long conversation, the context window limits what the AI can use from earlier in the exchange. A detail must be included in the input the model receives, either in the original messages or in a summary, for the model to refer back to it.
The same applies to uploaded materials: a long document takes up a large share of the window, so attaching several of them leaves less room for the conversation itself.
What Flint does when a session runs long
When a chat or session gets long, Flint automatically summarizes older messages before the model's context window fills up. Sparky receives that summary alongside recent messages, so the conversation can continue. Summarizing changes what the AI receives; it does not delete the original messages from your chat history.
A summary can leave out details. If Sparky loses an exact instruction, quotation or list from earlier in the conversation, repeat it or point Sparky to the attached source.
Two things help Flint keep the thread across sessions as well:
- Materials you attach to an activity or class are available to Sparky in every session, so they don't depend on the conversation history. See Uploading content to Flint.
- Memories let Sparky carry a small number of durable facts about a person between conversations. See Memories in Flint.
Key points to remember
- The context window is how much of a conversation a model can take into account at once.
- Its size is set by the model and measured in tokens.
- Attached documents share the window with the conversation.
- Flint summarizes older messages before the window fills up; summaries can omit details, so repeat important wording if needed. Memories carry facts between conversations.