The Context Window
The context window is the maximum amount of text a model can consider at once.
The context window is the total number of tokens the model can hold in view at one time — your instructions, the conversation history, any documents you paste in, and the reply it generates.
When a conversation grows past the limit, you must drop or summarise older messages. Bigger windows cost more and can be slower, so send only what the model needs.
Example
{
"model": "llm-medium",
"context_window_tokens": 128000,
"used": {"system": 40, "history": 3200, "documents": 6000, "reply_budget": 800},
"remaining": 117960
}When to use it
- A document-analysis tool chunks a 200-page PDF into 4k-token sections because the model's context window cannot hold the whole file.
- A customer-support agent truncates old messages from the conversation history to keep each API call within the context limit.
- A code-review bot measures the token count of a pull request diff before sending, falling back to a shorter summary if it is too large.
More examples
Check if text fits context window
Measures a document's token count against the model's context limit before sending it.
import tiktoken
MAX_CONTEXT = 128_000 # gpt-4o context limit
enc = tiktoken.encoding_for_model('gpt-4o')
def fits_context(text):
return len(enc.encode(text)) <= MAX_CONTEXT
with open('/tmp/big_document.txt') as f:
doc = f.read()
print('Fits context window:', fits_context(doc))Trim old messages to stay in limit
Removes oldest conversation turns until the total token count fits within the context window.
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o-mini')
LIMIT = 4096
def trim_messages(messages, limit=LIMIT):
while sum(len(enc.encode(m['content'])) for m in messages) > limit:
messages.pop(1) # drop oldest non-system message
return messagesSplit document into token-sized chunks
Slices a document into fixed-size token chunks so each piece fits within the context window.
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o-mini')
CHUNK_TOKENS = 2000
def chunk_text(text):
tokens = enc.encode(text)
return [
enc.decode(tokens[i:i + CHUNK_TOKENS])
for i in range(0, len(tokens), CHUNK_TOKENS)
]
with open('/tmp/article.txt') as f:
chunks = chunk_text(f.read())
print(f'Split into {len(chunks)} chunks')
Discussion