Observability for Agents on AWS
How to see inside a running agent using CloudWatch metrics, Bedrock model invocation logging, agent traces and a correlation id that ties them together.
Observability for an agent means being able to answer, after the fact and without redeploying anything, the question "what did this agent actually do for that user, and why?". It exists because an agent is not a normal service. A normal request goes in, one code path runs, one response comes out. An agent takes a goal, decides its own steps, calls tools in an order nobody wrote down, and produces a different trajectory on the next run with the same input. You cannot reason about that from an HTTP status code.
The unit of work is the session, not the request
A single user task may involve six model invocations, four tool calls and two retrieval queries. Ops dashboards that count requests and average latency will describe those as eleven healthy events. The user experienced one slow, expensive, wrong answer. Everything below exists to let you re-assemble that session.
Three layers, three AWS surfaces
- Metrics — CloudWatch. Amazon Bedrock publishes to the
AWS/Bedrocknamespace:Invocations,InvocationLatency,InputTokenCount,OutputTokenCount,InvocationClientErrors,InvocationServerErrorsandInvocationThrottles, dimensioned by model. Guardrails publish their own metrics, including counts of invocations where the guardrail intervened. These are cheap, aggregate and near real time — good for alarms, useless for diagnosis. - Logs — Bedrock model invocation logging. A single account-and-region setting (
PutModelInvocationLoggingConfiguration) that delivers the full request and response of every model invocation to CloudWatch Logs, Amazon S3, or both, with oversized payloads spilled to S3. It is off by default. Without it you have token counts but no prompts, and you cannot reconstruct why the model said what it said. - Traces — the agent's own reasoning. Calling
InvokeAgentwithenableTrace=Truestreams back trace events alongside the answer: pre-processing, orchestration (the model's rationale, eachinvocationInputto an action group or knowledge base, and eachobservationreturned), post-processing, guardrail and failure traces. For agents you host yourself, Amazon Bedrock AgentCore Observability emits OpenTelemetry spans into CloudWatch, and AWS X-Ray covers the downstream Lambda functions and APIs your tools call.
Correlation ids are the join key
None of these layers is useful alone. Pick one id per user task — the agent sessionId is the natural choice — and stamp it on everything: structured application logs, the payload you pass to every tool Lambda, the X-Ray annotation, the row you write to your evaluation store. Emit logs as JSON so CloudWatch Logs Insights can filter on it. When someone pastes a support ticket into Slack, one query on that id should return the whole trajectory.
What observability is not
It is not evaluation. CloudWatch will happily tell you the agent answered in 3.2 seconds using 12,000 tokens and zero errors, while the answer was confidently wrong. Metrics measure the machine; evaluation measures the work. It is also not free retention: trace events are streamed to your caller and vanish unless you write them somewhere, so treat persisting traces as part of the feature, not an afterthought.
The trade-off
Invocation logging captures raw prompts and completions, which for a support agent means customer PII sitting in a log group. Scope the log group's retention, encrypt it with your own KMS key, restrict read access, and consider S3 delivery with lifecycle rules rather than indefinite CloudWatch retention. Full-fidelity logging on a high-volume agent can also cost more than the inference it is observing — sample non-error sessions if that bites.
Example
import boto3, json
bedrock = boto3.client("bedrock")
# 1. Account- and region-level: turn on model invocation logging once.
bedrock.put_model_invocation_logging_configuration(
loggingConfig={
"cloudWatchConfig": {
"logGroupName": "/aws/bedrock/modelinvocations",
"roleArn": "arn:aws:iam::123456789012:role/BedrockLoggingRole",
"largeDataDeliveryS3Config": {
"bucketName": "acme-bedrock-logs",
"keyPrefix": "large-payloads/",
},
},
"textDataDeliveryEnabled": True,
"imageDataDeliveryEnabled": False,
"embeddingDataDeliveryEnabled": False,
}
)
# 2. Per-request: ask the agent for its trace and key everything on one id.
runtime = boto3.client("bedrock-agent-runtime")
session_id = "case-8842" # your correlation id for the whole task
response = runtime.invoke_agent(
agentId="AGENT1234567",
agentAliasId="ALIAS1234567",
sessionId=session_id,
enableTrace=True,
inputText="Why was invoice 4471 rejected?",
)
for event in response["completion"]:
if "trace" in event:
orch = event["trace"]["trace"].get("orchestrationTrace", {})
if "rationale" in orch:
print(json.dumps({"sessionId": session_id, "kind": "plan",
"text": orch["rationale"]["text"]}))
if "invocationInput" in orch:
print(json.dumps({"sessionId": session_id, "kind": "tool_call",
"detail": orch["invocationInput"]}, default=str))
if "observation" in orch:
print(json.dumps({"sessionId": session_id, "kind": "observation",
"detail": orch["observation"]}, default=str))
elif "chunk" in event:
print(event["chunk"]["bytes"].decode())When to use it
- A support agent starts giving wrong refund amounts; an engineer pastes the ticket's sessionId into CloudWatch Logs Insights and pulls the full trajectory — rationale, tool inputs and retrieved passages — in one query.
- A compliance team must show which prompts and completions were exchanged with a foundation model during an incident window, so invocation logs are delivered to an encrypted S3 bucket with a fixed retention policy.
- A platform team wires AgentCore Observability's OpenTelemetry spans into CloudWatch so self-hosted agents appear on the same dashboards as managed Bedrock agents.
Discussion