Context window in GenAI is the maximum number of tokens that a generative AI model can process and actively consider at a given time. It acts as a temporary buffer for recent interactions, but it does not store information permanently.
The context window includes the following:
- Space (in number of tokens) taken up by prior prompts and completions
- Space taken up by the current prompt
- Space left over for the upcoming completion
Note 1: Some LLMs use the above definition. Others include 1 and 2 in context window size and allocate separate space for 3.
Note 2: Tokens are the smallest units of information that the model understands, and the space within the context window is measured in tokens.
Note 3: Prompt is a piece of text that guides the generative AI model in generating its response. It can be a question, a statement, a description of a task, or anything else that provides the model with the necessary information to generate a relevant and informative response. The prompt can also set the tone or style of the model’s response.
For example, if you are having a conversation with a chatbot and you ask it a question, the chatbot will use the context window to generate its response. The context window will include the question itself, as well as any previous questions and answers in the conversation. This allows the chatbot to generate a response that is relevant to the current topic of discussion.
Example chat to illustrate context window which is comprised of all of the text shown in italics:
Prompt:
What is the capital of France?
Response:
Paris
Prompt:
What is the population of Paris?
Response:
The population of Paris is approximately 2.1 million people.
Without the context window, the chatbot would not know that the user was asking about the population of Paris. It might simply respond with a general fact about France. However, with the context window, the chatbot can generate a more specific and informative response.
Context windows are an important part of GenAI models. They allow the models to generate more coherent and informative responses, which makes them more useful in a variety of applications.
See Also: Prompt, Completion, Context