Controlling Agent Cost on Bedrock
Prompt caching, model right-sizing, distillation, batch inference and hard caps on loops and token budgets — and what each one costs you in return.
Agents are expensive in a way single-shot chat is not. Every step re-sends the whole conversation, so a five-step session with a 6,000-token system prompt and tool schemas pays for that prefix five times. Cost control is therefore mostly about not re-paying for the same tokens, and about stopping the agent from taking steps it did not need.
Prompt caching
Bedrock prompt caching lets you mark a point in the request with a cachePoint block; everything before it can be served from cache on subsequent calls that share an identical prefix. Cache writes cost more than ordinary input tokens and cache reads cost dramatically less, so caching pays off from the second hit onwards. Caches have a short time to live measured in minutes, and models impose a minimum prompt length before caching engages. Put the stable material first — system prompt, tool schemas, long policy documents, few-shot examples — and the volatile material after the cache point. The response usage block reports cacheReadInputTokens and cacheWriteInputTokens; if those stay at zero, your prefix is not stable and you are paying the write premium for nothing.
Right-sizing the model
Most agent steps are not hard. Classifying intent, extracting an order number, filling tool arguments and summarising a tool result can usually be done by the smallest model in a family; only planning and final synthesis need the large one. Route per step rather than picking one model for the whole agent. Bedrock's intelligent prompt routing can make that choice for you within a model family based on predicted response quality. Measure quality per task type before you switch, not overall — a small model that is fine at extraction can be hopeless at multi-step planning.
Distillation
Amazon Bedrock Model Distillation trains a smaller student model on responses generated by a larger teacher, optionally using prompts harvested from your own model invocation logs. You get near-teacher quality on your narrow task at student-model cost and latency. The catch: a distilled model is a custom model, and you serve it through Provisioned Throughput, which you pay for by the hour whether or not traffic arrives. Distillation is a volume play — it only makes sense when the workload is steady, high-volume and narrow enough that the student generalises.
Batch what does not need to be interactive
Bedrock batch inference (CreateModelInvocationJob) reads a JSONL manifest from S3, processes it asynchronously at a discount to on-demand pricing, and writes results back to S3. Turnaround is hours, so it is wrong for a user waiting on a chat reply and right for everything around the agent: nightly document enrichment, embedding backfills, bulk classification, and running your evaluation suite.
Hard caps are not optional
A looping agent is an unbounded bill. Enforce a maximum step count, a maximum token budget per session, and a wall-clock deadline in the orchestration loop itself, and decide in advance what happens when a cap is hit — hand off to a human, return partial results, ask a clarifying question. Never let the cap surface as a stack trace. Also cap maxTokens per call; agents that ramble produce long outputs that then become long inputs on the next step.
The trade-off
Every lever here trades against something. Caching constrains how you build prompts. Small models raise retry rates. Distillation buys hourly capacity. Caps truncate genuinely hard tasks. The cheapest token is still the one you never send: if the task has a fixed sequence of steps, a Step Functions workflow calling the model twice will beat an agent that reasons its way to the same place at ten times the cost.
Example
import boto3
client = boto3.client("bedrock-runtime")
STABLE_PREFIX = open("system_prompt.md").read() # byte-identical every call
MAX_STEPS = 6
TOKEN_BUDGET = 60_000
messages = [{"role": "user", "content": [{"text": "Refund order 4471 if eligible."}]}]
spent = 0
for step in range(MAX_STEPS):
resp = client.converse(
modelId="arn:aws:bedrock:eu-west-1:123456789012:application-inference-profile/support-agent",
system=[
{"text": STABLE_PREFIX},
{"cachePoint": {"type": "default"}}, # everything above is cacheable
],
toolConfig={"tools": TOOLS},
messages=messages,
inferenceConfig={"maxTokens": 800, "temperature": 0},
)
u = resp["usage"]
spent += u["inputTokens"] + u["outputTokens"]
print("cache read:", u.get("cacheReadInputTokens", 0),
"cache write:", u.get("cacheWriteInputTokens", 0),
"budget used:", spent)
messages.append(resp["output"]["message"])
if resp["stopReason"] != "tool_use":
break
if spent > TOKEN_BUDGET:
escalate_to_human(reason="token budget exhausted") # graceful, not a crash
break
messages.append(run_tools(resp["output"]["message"]))When to use it
- A document-QA agent moves its 9,000-token policy manual and tool schemas above a cache point, cutting the per-session bill sharply because the prefix is re-read on every loop step.
- A team distils a large model into a small one for a high-volume invoice-classification agent, accepting Provisioned Throughput's hourly commitment because the workload runs continuously.
- Nightly re-summarisation of the previous day's tickets moves from live InvokeModel calls to a Bedrock batch inference job reading and writing S3.
Discussion