Observability & Deployment Health Gates
A deploy isn't 'done' when the task is running — it's done when the metrics say it's healthy. Gate rollouts on real signals, not on exit code 0.
Junior pipelines celebrate when update-service returns success. Senior pipelines know that's just the start of the verification window. The question isn't "did it deploy?" — it's "is the new version actually healthier than the old one?"
Layer your health signals
- Liveness — is the process up? (systemd / container running)
- Readiness —
GET /healthzreturns 200 only when dependencies (DB, cache) are reachable. The ALB routes on this. - Golden signals — error rate, latency (p99), traffic, saturation, tracked in CloudWatch and alarmed.
Gate the rollout on the signal
Turn those signals into a gate: during a canary window, a CloudWatch alarm on 5xx or p99 latency is wired into the deployment so a breach aborts and rolls back automatically. CodeDeploy lifecycle hooks (AfterAllowTestTraffic) can even run a Lambda that fires synthetic requests before real users see the new version.
If you can't observe a release, you can't safely automate it. Dashboards and alarms are deploy infrastructure, not an afterthought.
Example
// CloudWatch alarm used as a deployment health gate
{
"AlarmName": "myapp-prod-5xx-high",
"Namespace": "AWS/ApplicationELB",
"MetricName": "HTTPCode_Target_5XX_Count",
"Dimensions": [
{ "Name": "LoadBalancer", "Value": "app/myapp-prod/abc123" }
],
"Statistic": "Sum",
"Period": 60,
"EvaluationPeriods": 2,
"Threshold": 10,
"ComparisonOperator": "GreaterThanThreshold",
"TreatMissingData": "notBreaching",
"AlarmActions": [
"arn:aws:sns:us-east-1:123456789012:deploy-alerts"
]
}When to use it
- A team waits 5 minutes after each canary traffic shift and checks a CloudWatch alarm before proceeding, so a latency regression stops the rollout automatically.
- An ECS rolling deploy pauses until ALB health checks report all new tasks healthy before terminating any old tasks.
- A CI pipeline queries CloudWatch Logs Insights for errors in the 10 minutes after deploy and fails the pipeline if the error rate exceeds a threshold.
More examples
Wait for ECS service stability
Waits for ECS tasks to stabilise then checks the error alarm state — if it is not OK the script aborts the deployment.
aws ecs wait services-stable \
--cluster prod \
--services myapp
# Then check the CloudWatch alarm
STATE=$(aws cloudwatch describe-alarms \
--alarm-names myapp-error-rate-high \
--query 'MetricAlarms[0].StateValue' \
--output text)
[ "$STATE" = 'OK' ] && echo 'Health gate passed' || { echo 'ALARM — aborting'; exit 1; }CloudWatch Logs error gate
Queries log errors in the post-deploy window and fails the pipeline if more than 5 errors appear, acting as a log-based health gate.
QUERY_ID=$(aws logs start-query \
--log-group-name /ecs/myapp \
--start-time $(date -d '5 min ago' +%s) \
--end-time $(date +%s) \
--query-string 'filter @message like /ERROR/ | stats count() as errors' \
--query 'queryId' --output text)
sleep 10
ERRORS=$(aws logs get-query-results --query-id $QUERY_ID \
--query 'results[0][0].value' --output text)
[ "${ERRORS:-0}" -lt 5 ] && echo 'Gate passed' || { echo 'Too many errors'; exit 1; }Composite health check in deploy script
Polls the /health endpoint up to 6 times with 10-second intervals and fails the deploy if it never returns 200.
check_health() {
local ENDPOINT=$1
for i in {1..6}; do
HTTP=$(curl -s -o /dev/null -w '%{http_code}' $ENDPOINT/health)
[ "$HTTP" = '200' ] && return 0
echo "Attempt $i: got $HTTP, retrying..."
sleep 10
done
return 1
}
check_health https://api.example.com || { echo 'Health gate failed'; exit 1; }
Discussion