Logging & Observability that Actually Helps
Go beyond raw awslogs — structure your logs, enable Container Insights, and know which signals matter during an incident.
Everyone ships logs to CloudWatch with the awslogs driver and calls it a day. That is table stakes. The difference at 3am, when a service is misbehaving, is whether those logs and metrics are actually usable.
Structure your logs
Emit JSON, not free-form text. Structured logs let you use CloudWatch Logs Insights to filter by requestId, latency, or status code instead of grepping. A single consistent schema across services pays for itself the first time you have to correlate a request across three of them.
Enable Container Insights
Cluster-level CPU averages hide the task that is quietly melting. Container Insights gives per-task CPU, memory, network, and storage, plus curated dashboards. Turn it on per cluster — the small cost is worth it the moment you need it.
Know your first three stops in an incident
describe-services→ the events array. It plainly states why tasks will not place or register.- Container Insights → the offending task's memory/CPU curve.
- CloudWatch Logs Insights → the structured error, filtered to the failing task.
Observability is not something you bolt on during an outage. Wire it in while things are calm, so that when they are not, the answer is already waiting for you.
Example
{
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/web",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "web",
"awslogs-datetime-format": "%Y-%m-%dT%H:%M:%S",
"mode": "non-blocking",
"max-buffer-size": "4m"
}
}
}When to use it
- A team switches from unstructured log lines to JSON-formatted logs so CloudWatch Logs Insights can query fields like requestId and statusCode directly.
- An SRE creates a CloudWatch Logs Insights query that finds all ERROR-level log entries across all tasks in the last hour for a nightly reliability report.
- A developer uses Container Insights task-level metrics to confirm that memory usage is growing linearly over 24 hours, confirming a slow memory leak.
More examples
JSON-Structured App Log Example
Shows what a structured JSON log line looks like so CloudWatch Logs Insights can filter and aggregate on individual fields like statusCode or durationMs.
{
"timestamp": "2024-01-15T10:23:45Z",
"level": "INFO",
"requestId": "a1b2c3d4",
"method": "GET",
"path": "/api/orders",
"statusCode": 200,
"durationMs": 42
}Logs Insights Query for Errors
Runs a Logs Insights query to find the 50 most recent ERROR log events in the last hour, useful for rapid post-incident analysis.
aws logs start-query \
--log-group-name /ecs/api \
--start-time $(date -d '-1 hour' +%s) \
--end-time $(date +%s) \
--query-string 'fields @timestamp, level, requestId, @message
| filter level = "ERROR"
| sort @timestamp desc
| limit 50'CloudWatch Dashboard for ECS Service
Creates a CloudWatch dashboard showing CPU utilization, memory utilization, and running task count for the ECS service in one view.
aws cloudwatch put-dashboard \
--dashboard-name ECS-Api-Service \
--dashboard-body '{
"widgets": [
{"type":"metric","properties":{"metrics":[["ECS/ContainerInsights","CpuUtilized","ClusterName","prod","ServiceName","api"],[".","MemoryUtilized",".",".",".","."]],"title":"CPU & Memory","period":60}},
{"type":"metric","properties":{"metrics":[["AWS/ECS","RunningTaskCount","ClusterName","prod","ServiceName","api"]],"title":"Running Tasks","period":60}}
]
}'
Discussion