Cutting Agent Latency
Streaming, parallel tool calls and retrieval tuning attack different parts of an agent's latency budget — and only one of them makes the total time smaller.
Agent latency is additive. Total time is the sum of every model invocation, every tool call and every retrieval in the loop, and each extra step adds a full model round trip over a prompt that has grown since the last one. Before optimising anything, break the budget down from your traces: how much is model time, how much is tool time, how much is retrieval, and how many steps ran. Teams routinely tune the model when a cold-starting Lambda tool or a six-step loop owns the wall clock.
Streaming fixes perception, not duration
With ConverseStream or InvokeModelWithResponseStream, tokens arrive as they are generated, so the user sees text in a few hundred milliseconds instead of staring at a spinner. Bedrock agents can stream their final response too. Total time to the last token is unchanged — streaming buys you time-to-first-token, which is what users actually judge. Two caveats: a streamed response has to be guarded incrementally, so guardrails are applied to buffered chunks of output rather than to one complete string, which means text can be shown and then withdrawn; and you cannot stream the intermediate steps of a loop as an answer, only narrate them.
Parallel tool calls shrink real time
When a model returns several toolUse blocks in one response, those calls are independent by construction — run them concurrently and return all the toolResult blocks in a single follow-up message. Three 800 ms lookups become one 800 ms step instead of three sequential ones, and crucially it is one model round trip rather than three. Design tools so parallelism is possible: a get_customer and a get_orders tool can run together, while a search that must feed a fetch cannot. Set a per-tool timeout, because the step is as slow as its slowest call.
Retrieval tuning
Retrieval hurts twice: the vector query itself, and the tokens it drops into a prompt you then pay to process on every subsequent step. Lower numberOfResults to the smallest number that keeps answers correct. Use metadata filters so the search space is a slice, not the whole index. Compare hybrid against pure semantic search — hybrid often lets you retrieve fewer chunks for the same recall. Reranking adds a network hop but frequently pays for itself by letting you send three excellent chunks instead of ten mediocre ones. Chunk size matters in both directions: chunks that are too small force you to fetch more of them, chunks that are too big waste context on irrelevant text.
Model-side levers
Prompt length drives time to first token, so the prompt-caching work you do for cost also helps latency. Bedrock offers latency-optimised inference on selected models via a performance configuration on the request. Cross-region inference profiles route around regional capacity pressure and reduce throttling, which is often what a latency spike really is — check InvocationThrottles before blaming the model.
The failure mode
The largest single latency lever is usually step count, and step count is a prompt and tool-design problem, not an infrastructure one. An agent that asks three clarifying questions because its instructions are vague, or that retries a tool because the error message was unhelpful, cannot be optimised into speed. Fix the loop first; then stream what remains.
Example
import time, boto3
from concurrent.futures import ThreadPoolExecutor
client = boto3.client("bedrock-runtime")
start = time.monotonic()
ttft = None
stream = client.converse_stream(
modelId="eu.anthropic.claude-sonnet-4-5-20250929-v1:0",
system=[{"text": SYSTEM}, {"cachePoint": {"type": "default"}}],
toolConfig={"tools": TOOLS},
messages=messages,
inferenceConfig={"maxTokens": 700, "temperature": 0},
)
tool_uses = []
for event in stream["stream"]:
if "contentBlockDelta" in event:
if ttft is None:
ttft = (time.monotonic() - start) * 1000 # emit as a metric
delta = event["contentBlockDelta"]["delta"]
if "text" in delta:
print(delta["text"], end="", flush=True)
elif "contentBlockStop" in event:
pass
elif "messageStop" in event:
stop_reason = event["messageStop"]["stopReason"]
# The model asked for several tools at once: run them together, not in sequence.
def call(tool_use):
return {
"toolResult": {
"toolUseId": tool_use["toolUseId"],
"content": [{"json": DISPATCH[tool_use["name"]](tool_use["input"])}],
}
}
if stop_reason == "tool_use":
with ThreadPoolExecutor(max_workers=4) as pool:
results = list(pool.map(call, tool_uses))
# All results go back in ONE user message = one extra model round trip.
messages.append({"role": "user", "content": results})When to use it
- A travel agent fetches flights, hotels and the customer's loyalty status concurrently, turning a three-second step into a one-second one.
- A chat assistant switches to ConverseStream so users see the first sentence in under a second, even though the full answer still takes six.
- A retrieval-heavy agent adds a reranker and drops from ten retrieved chunks to three, cutting both prompt-processing time and per-step cost.
Discussion