Summarising Older Turns, Keeping Recent Turns Verbatim
Compaction keeps the last few turns byte-exact and replaces everything older with a structured running summary, trading detail for headroom.
When a conversation outgrows its token budget you have three options: truncate, summarise, or externalise. Truncation is the worst - dropping the oldest turns silently discards the constraints the user set at the start. The standard answer is a hybrid window: keep the most recent turns verbatim and compact everything older into a running summary block.
The asymmetry is the whole idea. Recent turns matter as wording, because that is what pronouns, ordinals and corrections refer to. Older turns matter as facts - what was decided, which entity we are talking about, what the user ruled out. Facts compress well; wording does not.
How to do it
Trigger on tokens, not turn count. Turns vary wildly in size - one tool result can be larger than twenty exchanges - so count tokens after each turn and compact when you cross a threshold, well before the hard limit.
When the threshold trips, take the oldest chunk of history, send it to a cheaper, faster model on Bedrock with a summarisation prompt, and splice the result back in as a single block ahead of the retained turns. Ask for structured output rather than prose: decisions made, entities and identifiers in play, constraints stated, open questions, tool outcomes. A schema makes the summary stable across generations, mergeable with the previous summary, and far less likely to invent details.
Pin the things that must never be lost into a separate block that is never summarised. A constraint stated once at turn two - an allergy, a hard budget ceiling, a jurisdiction - should be extracted and held verbatim, not entrusted to a chain of summaries.
The tool-use trap
This is where implementations break. In the Converse API a message containing a toolUse block must be followed by a message containing the matching toolResult. If your cut lands between them, you have orphaned the pair and the request fails validation. Compaction must snap to a clean boundary: cut only immediately before a user turn that is not a tool result. Get this wrong and it works fine in testing and fails the first time a real conversation compacts mid-tool-call.
The trade-offs
Summarisation is another model call, so it costs money and adds latency to the turn that triggers it - do it asynchronously between turns where you can. It is lossy, and errors compound: a summary of a summary of a summary drifts. And it invalidates prompt caching, because you have just rewritten the prefix that was cached; place the summary block so that the compaction rewrites as little of the cacheable prefix as possible.
There is often a better alternative. Keep the complete transcript in DynamoDB or S3 as the system of record and give the agent a search_conversation_history tool. Instead of carrying every old turn, the agent pulls back the one it needs, when it needs it. You pay retrieval latency occasionally rather than token cost on every turn - and nothing is lost to summarisation drift.
Example
import json
SUMMARY_SCHEMA_PROMPT = """Summarise the conversation below as JSON with keys:
decisions, entities, constraints, open_questions, tool_outcomes.
Record only what is stated. Do not infer. Keep identifiers verbatim."""
def safe_cut(messages, keep_recent):
"""Find a cut index that never orphans a toolUse/toolResult pair."""
idx = max(0, len(messages) - keep_recent)
while idx < len(messages):
msg = messages[idx]
is_tool_result = any("toolResult" in b for b in msg["content"])
if msg["role"] == "user" and not is_tool_result:
return idx # clean boundary
idx += 1
return len(messages)
def compact(messages, running_summary, keep_recent=8):
cut = safe_cut(messages, keep_recent)
if cut == 0:
return messages, running_summary
older, recent = messages[:cut], messages[cut:]
result = bedrock.converse(
modelId=SUMMARISER_MODEL_ID, # a smaller, cheaper model
system=[{"text": SUMMARY_SCHEMA_PROMPT}],
messages=[{"role": "user", "content": [{"text": render(running_summary, older)}]}],
inferenceConfig={"maxTokens": 800, "temperature": 0},
)
new_summary = json.loads(result["output"]["message"]["content"][0]["text"])
summary_block = {"text": "<conversation_summary>"
+ json.dumps(new_summary)
+ "</conversation_summary>"}
# recent[0] is a user turn by construction, so the summary cannot be a
# message of its own: Converse rejects two consecutive user messages.
# Merge it into the first retained user turn instead.
if recent:
head = {"role": "user", "content": [summary_block] + recent[0]["content"]}
compacted = [head] + recent[1:]
else:
compacted = [{"role": "user", "content": [summary_block]}]
return compacted, new_summaryWhen to use it
- A long-running incident-response agent that has been iterating with an engineer for two hours and must keep the last few exchanges exact while compressing the earlier triage.
- A travel booking agent that summarises the search phase but keeps the last five turns verbatim so 'the second flight' still resolves.
- A team that replaces summarisation entirely with a search_conversation_history tool backed by DynamoDB, because their agent needs exact quotes from earlier turns.
Discussion