What to Measure: Agent SLIs That Matter
The specific signals — tokens, time to first token, tool latency, loop length, guardrail interventions and cost per resolved task — that tell you whether an agent is healthy.
Once you can see inside an agent, you have to decide what is worth watching. Classic service indicators — rate, errors, duration — still apply, but they miss every failure mode that is specific to agents. An agent can return HTTP 200 in 900 milliseconds after taking nine steps, burning 40,000 tokens and giving up. The metrics below exist because that session must show up as unhealthy on a dashboard.
The core set
| Signal | Where it comes from | What it tells you |
|---|---|---|
| Token consumption per session | usage block on Converse responses, or InputTokenCount/OutputTokenCount in CloudWatch | Your bill, and prompt bloat. Split out cache-read and cache-write tokens or caching wins will be invisible. |
| Time to first token (TTFT) | Timestamp of the first streamed chunk minus request start | What the user perceives as responsiveness. Only meaningful when you stream. |
| Tool latency, per tool | Lambda duration, X-Ray subsegments, your own timers | Which dependency owns the wall clock. Track p95, not the mean. |
| Loop length (steps per session) | Count of orchestration turns in the trace | The best early-warning metric there is. Rising loop length precedes cost blowouts, timeouts and quality collapse. |
| Guardrail intervention rate | Guardrail CloudWatch metrics and guardrail trace events | Whether inputs are turning hostile, or your guardrail is over-blocking legitimate users. |
| Cost per resolved task | Session cost divided by a business outcome signal | The only number a finance or product owner cares about. |
Cost per resolved task deserves special care
Cost per invocation is easy and misleading: making the model cheaper and dumber lowers it while raising the number of attempts a user needs. You need a resolution signal — ticket closed without human escalation, order actually placed, user did not re-ask within an hour — emitted as a metric on the same session id. Cost per resolved task then falls when the agent gets genuinely better and rises when it gets cheap and useless. Attribute the cost side with Bedrock application inference profiles, which let you tag invocations so spend shows up per application in Cost Explorer.
Aggregate per session, alarm on percentiles
Compute your indicators at session granularity, then publish them with the CloudWatch embedded metric format (EMF): write one structured log line per session and CloudWatch extracts the metrics from it. Keep dimensions low-cardinality — environment, agent alias, tool name — and leave sessionId and traceId as ordinary log fields you can query but not as dimensions, or you will create a metric per user.
Averages are actively harmful here. The distribution of agent sessions is long-tailed: most finish in two steps, a few grind through fifteen. A mean loop length of 2.4 hides the sessions that are costing you real money. Alarm on p95 loop length, p95 TTFT and p99 tokens per session.
What this is not
None of these is a quality metric. A low guardrail intervention rate is not safety, and a short loop is not correctness — an agent that immediately answers "I can't help with that" scores beautifully on every indicator here. Operational metrics tell you the system is behaving as designed; only evaluation tells you the design is right. Run both.
Example
{
"_aws": {
"Timestamp": 1756180000000,
"CloudWatchMetrics": [
{
"Namespace": "Acme/SupportAgent",
"Dimensions": [["Environment", "AgentAlias"]],
"Metrics": [
{ "Name": "SessionInputTokens", "Unit": "Count" },
{ "Name": "SessionOutputTokens", "Unit": "Count" },
{ "Name": "CacheReadTokens", "Unit": "Count" },
{ "Name": "TimeToFirstToken", "Unit": "Milliseconds" },
{ "Name": "LoopSteps", "Unit": "Count" },
{ "Name": "GuardrailIntervened", "Unit": "Count" },
{ "Name": "TaskResolved", "Unit": "Count" },
{ "Name": "SessionCostUsd", "Unit": "None" }
]
}
]
},
"Environment": "prod",
"AgentAlias": "support-v7",
"sessionId": "case-8842",
"traceId": "1-68ad4f21-4c9b2a7e",
"slowestTool": "lookup_invoice",
"slowestToolMs": 2140,
"SessionInputTokens": 18422,
"SessionOutputTokens": 1105,
"CacheReadTokens": 15980,
"TimeToFirstToken": 780,
"LoopSteps": 4,
"GuardrailIntervened": 0,
"TaskResolved": 1,
"SessionCostUsd": 0.0143
}When to use it
- An alarm on p95 loop length fires after a prompt change, catching an agent that started re-calling the same search tool three times per session before the monthly bill revealed it.
- A product owner tracks cost per resolved ticket week over week to justify moving from a large model to a smaller one for triage.
- A safety team watches guardrail intervention rate per agent alias and spots an over-tuned denied-topic policy blocking legitimate billing questions.
Discussion