Troubleshooting Stuck Deployments
A calm, repeatable playbook for the deploy that just will not go green — from PENDING tasks to failing health checks.
Sooner or later a deployment hangs: the new revision will not stabilize, old tasks will not drain, and the service sits at 1 of 2 running. Panic is optional. There is a short list of causes, and a methodical sweep finds it almost every time.
Start where ECS is already telling you
The service events from describe-services are the most under-used diagnostic in ECS. They say, in plain English, "unable to place task because no container instance met requirements" or "task failed ELB health checks." Read them first, always.
Match the symptom to the cause
| Symptom | Usual cause |
|---|---|
Task stuck in PROVISIONING | No free IPs in the subnet, or ENI limit reached |
Task stuck in PENDING, then stops | Cannot pull image or fetch secret (execution role / networking) |
| Task starts then is killed | Failing health check, or OOM (memory too low) |
| New tasks never register | Security group blocks the ALB, or wrong container port |
Read the stopped task
A stopped task carries a stoppedReason and per-container exitCode. Exit code 137 means it was OOM-killed — give it more memory. A CannotPullContainerError points at ECR access or missing NAT/VPC endpoints. These two fields resolve the majority of failures.
The answer is almost never hidden. Service events plus the stopped task's reason tell you what happened in over 90% of stuck deploys — if you slow down and actually read them.
Example
# 1. Read the service events first — they usually name the problem.
aws ecs describe-services --cluster prod --services web \
--query 'services[0].events[0:5].message'
# 2. Inspect the most recent STOPPED task for the real reason.
TASK=$(aws ecs list-tasks --cluster prod --service-name web \
--desired-status STOPPED --query 'taskArns[0]' --output text)
aws ecs describe-tasks --cluster prod --tasks "$TASK" \
--query 'tasks[0].{reason:stoppedReason,containers:containers[].{name:name,exit:exitCode,reason:reason}}'When to use it
- An on-call engineer follows a repeatable troubleshooting playbook when a service deployment stalls at PENDING, starting with describe-tasks to read the stop reason.
- A developer checks CloudWatch Logs for the stoppedReason field to discover that a task failed because Secrets Manager denied the execution role access to a secret.
- An SRE uses the deployment circuit breaker rollback event in CloudTrail to find the exact timestamp and reason a bad deployment was automatically reverted.
More examples
Describe Stopped Task Stop Reason
Retrieves the top-level stop reason and per-container exit codes for a stopped task — the first step in diagnosing any failed deployment.
aws ecs describe-tasks \
--cluster prod \
--tasks arn:aws:ecs:us-east-1:123456789012:task/prod/abc123 \
--query 'tasks[0].{status:lastStatus,stopped:stoppedReason,containers:containers[*].{name:name,exit:exitCode,reason:reason}}'Check Service Events for Errors
Lists the 10 most recent service events chronologically, revealing registration failures, deregistration from the ALB, and circuit breaker triggers.
aws ecs describe-services \
--cluster prod \
--services api \
--query 'services[0].events[:10].{created:createdAt,message:message}' \
--output tableFind Tasks Stuck in PENDING State
Lists all PENDING tasks and describes their state and reason, quickly revealing resource-exhaustion, image-pull, or networking issues blocking task startup.
PENDING=$(aws ecs list-tasks \
--cluster prod \
--desired-status PENDING \
--query 'taskArns[]' --output text)
if [ -n "$PENDING" ]; then
aws ecs describe-tasks \
--cluster prod \
--tasks $PENDING \
--query 'tasks[*].{arn:taskArn,status:lastStatus,reason:stoppedReason}'
fi
Discussion