The Context Window as a Budget
The context window is a fixed per-request token budget shared by everything you send and everything the model generates - not a memory store.
The context window is the maximum number of tokens a model can attend to in a single request. It is a per-call buffer, not storage. Every request to Amazon Bedrock is stateless: the model sees exactly what is in that one payload and nothing else. When the response comes back, the window is gone. If you want the model to recall turn three, you must send turn three again.
This is the distinction the exam keeps returning to: context is not memory. Memory is what you keep in a store. Context is what you paid to put in front of the model on this call.
What actually fills the budget
In a toy chatbot the conversation is the whole payload. In an agent it is usually the smallest part. A typical agent turn carries:
- The system prompt or agent instruction.
- Tool and action-group schemas - every parameter description is tokens, on every turn.
- Knowledge base retrieval results, if you are doing RAG.
- The conversation transcript so far.
- Tool results - usually the largest and most underestimated item. One unfiltered API response or SQL result set can dwarf the entire conversation.
- Headroom for the model's output, because input and output share the same window.
What it costs
Bedrock bills separately for input tokens and output tokens. Because the whole transcript is resent every turn, the cost of an N-turn conversation does not grow linearly with N - it grows roughly with the square of it. Turn ten pays for turns one to nine all over again. Latency follows the same curve: time to first token scales with how much input the model must process.
Prompt caching is the direct mitigation for the stable part of that payload. In the Converse API you insert a cachePoint block after content that does not change between calls - the system prompt and tool schemas, typically - and Bedrock can serve that prefix from cache at a reduced rate. Caching is prefix-based, so anything volatile (a timestamp, a per-request ID) placed before the cache point invalidates everything after it.
The failure modes
Exceeding the window is the obvious one: Bedrock returns a validation error rather than silently truncating, so an agent that never prunes will simply stop working mid-conversation. The subtler failure is quality. Models do not attend uniformly across a very long window; a constraint stated once, thousands of tokens back, is more likely to be missed than one stated recently. A 200K-token window is not 200K tokens of reliable recall.
Treat the window as a budget you allocate deliberately: a fixed reserve for instructions and tools, a bounded slice for retrieval, a bounded slice for history, and guaranteed room for the answer. Everything that does not fit that budget belongs in a store outside the model - which is what the rest of this chapter is about.
Example
import os
import boto3
bedrock = boto3.client("bedrock-runtime")
MODEL_ID = os.environ["MODEL_ID"] # inference profile or model id for your region
# Stable prefix first, then a cache point, then volatile content.
system = [
{"text": AGENT_INSTRUCTIONS}, # never changes
{"cachePoint": {"type": "default"}}, # everything above may be cached
]
response = bedrock.converse(
modelId=MODEL_ID,
system=system,
messages=messages, # rolling transcript
toolConfig={"tools": TOOL_SCHEMAS},
inferenceConfig={"maxTokens": 1024}, # reserve room for the answer
)
usage = response["usage"]
print(
"in:", usage["inputTokens"],
"out:", usage["outputTokens"],
"cache_read:", usage.get("cacheReadInputTokens", 0),
"cache_write:", usage.get("cacheWriteInputTokens", 0),
)
# Budget check before the next turn, not after the failure.
if usage["totalTokens"] > COMPACTION_THRESHOLD:
messages = compact(messages)When to use it
- A support agent whose per-conversation Bedrock cost triples between turn five and turn fifteen because the full transcript plus twelve action-group schemas is resent each time.
- An agent that queries CloudWatch Logs and pastes the raw result into context, blowing the window on a single tool call.
- A team enabling prompt caching on a large, stable system prompt so repeated turns bill the prefix at cache-read rates instead of full input rates.
Discussion