Monitoring & Observability with CloudWatch
Ship metrics, logs and traces into CloudWatch, alarm on the things users actually feel, and treat dashboards as the story you tell during an incident.
You cannot operate what you cannot see. Observability is the discipline of always being able to answer "is it healthy, and if not, why?" — and on AWS that starts with CloudWatch.
The three signals
- Metrics — numbers over time (CPU, latency, error count, queue depth). Cheap to store, fast to alarm on.
- Logs — the detailed narrative. Ship app and system logs to CloudWatch Logs and query them with Logs Insights.
- Traces — with X-Ray, follow a single request across services to find the slow hop.
Alarm on symptoms, not causes
Page a human for what the user experiences — high p99 latency, elevated 5xx rate, a growing dead-letter queue. High CPU alone is not an incident; slow responses are. Wire alarms to SNS so they reach a human (or PagerDuty), and use composite alarms to cut noise.
Custom metrics tell your story
The built-in metrics do not know about "orders per minute" or "checkout failures". Publish custom metrics (or better, structured Embedded Metric Format logs) so your dashboards speak in business terms, not just infrastructure.
Example
# Alarm when p99 latency crosses 1s for 3 straight minutes -> SNS page
aws cloudwatch put-metric-alarm \
--alarm-name web-p99-latency-high \
--namespace AWS/ApplicationELB \
--metric-name TargetResponseTime \
--extended-statistic p99 \
--dimensions Name=LoadBalancer,Value=app/web-alb/50dc6c495c0c9188 \
--period 60 --evaluation-periods 3 --threshold 1.0 \
--comparison-operator GreaterThanThreshold \
--treat-missing-data notBreaching \
--alarm-actions arn:aws:sns:us-east-1:123456789012:oncall-pager
# Top 20 slowest requests in the last hour via Logs Insights
aws logs start-query \
--log-group-name /acme/web \
--start-time $(date -d '1 hour ago' +%s) \
--end-time $(date +%s) \
--query-string 'fields @timestamp, @message | sort @duration desc | limit 20'When to use it
- An SRE team creates a CloudWatch dashboard with p99 latency, error rate, and RDS connection count so the on-call engineer has a single screen during incidents.
- A developer configures structured JSON logging in a Lambda function so CloudWatch Logs Insights can query fields like userId and errorCode across millions of events.
- Alerts in CloudWatch fire when the ALB 5xx error rate exceeds 1% for two minutes, triggering a PagerDuty page to the on-call engineer.
More examples
Create CloudWatch Dashboard
Creates a CloudWatch dashboard widget showing ALB 5xx errors per minute, giving the on-call engineer a real-time error view.
aws cloudwatch put-dashboard \
--dashboard-name AppOverview \
--dashboard-body '{
"widgets": [
{
"type": "metric",
"properties": {
"metrics": [["AWS/ApplicationELB","HTTPCode_ELB_5XX_Count","LoadBalancer","app/web-alb/abc"]],
"period": 60,
"title": "ALB 5xx Errors"
}
}
]
}'Set Alarm on Error Rate
Creates an alarm that triggers an SNS (PagerDuty) notification when ALB returns more than 50 errors per minute for two consecutive minutes.
aws cloudwatch put-metric-alarm \
--alarm-name alb-5xx-high \
--metric-name HTTPCode_ELB_5XX_Count \
--namespace AWS/ApplicationELB \
--dimensions Name=LoadBalancer,Value=app/web-alb/abc \
--statistic Sum \
--period 60 \
--threshold 50 \
--comparison-operator GreaterThanThreshold \
--evaluation-periods 2 \
--alarm-actions arn:aws:sns:us-east-1:123456789012:pagerduty-snsQuery Structured Lambda Logs
Runs a Logs Insights aggregation query that counts errors grouped by errorCode, identifying the top failure modes in the last 30 minutes.
aws logs start-query \
--log-group-name /aws/lambda/payment-service \
--start-time $(date -d '30 minutes ago' +%s) \
--end-time $(date +%s) \
--query-string 'fields @timestamp, level, errorCode, userId | filter level = "ERROR" | stats count(*) as errors by errorCode | sort errors desc'
Discussion