Designing for High Availability

Assume every component will eventually fail, then spread across Availability Zones, let Auto Scaling replace the dead, and wire load-balancer health checks so traffic only reaches healthy targets.

Werner Vogels' line says it best: everything fails, all the time. High availability is not about preventing failure — it is about failure not mattering. Three ingredients get you most of the way.

1. Spread across Availability Zones

A single AZ is a set of data centers that can lose power together. Run at least two AZs for anything that matters: multiple subnets, a Multi-AZ RDS deployment, an Auto Scaling group told to balance across zones. One AZ dies, you stay up.

2. Make instances disposable

An Auto Scaling group with a sensible minimum keeps replacing failed instances automatically. Combined with immutable, user-data-bootstrapped AMIs, a sick server is just terminated and reborn — no 3am SSH session required.

3. Health checks that tell the truth

Point your load balancer at a real health-check endpoint — ideally one that verifies dependencies (can it reach the database?), not just "is the port open?". Use ELB health checks on the ASG so a target that fails checks is drained and replaced, not left quietly serving errors.

The subtle failure mode: a shallow health check returns 200 while the app cannot reach its database. The LB keeps sending traffic into a black hole. A deep-but-cheap check catches this.

Example

# Target group with a DEEP health check on /healthz
aws elbv2 create-target-group \
  --name web-tg --protocol HTTP --port 8080 \
  --vpc-id vpc-0a1b2c3d4e5f67890 \
  --health-check-path /healthz \
  --health-check-interval-seconds 15 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 2

# ASG spanning two AZs, using ELB health so bad targets get replaced
aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name web-asg \
  --launch-template LaunchTemplateName=web-lt,Version='$Latest' \
  --min-size 2 --max-size 8 --desired-capacity 2 \
  --health-check-type ELB --health-check-grace-period 90 \
  --vpc-zone-identifier "subnet-0aaa111,subnet-0bbb222"

When to use it

An application deploys EC2 instances across three AZs in an Auto Scaling group so losing one entire data center does not cause downtime.
An RDS instance is configured with Multi-AZ so AWS automatically fails over to a synchronous standby replica within 60 seconds of a primary failure.
An S3-backed static site with CloudFront achieves 99.99% availability because both services replicate across multiple AZs and edge locations by default.

More examples

Deploy ASG Across Three AZs

Spreads a minimum of 3 EC2 instances across 3 AZs so the application survives the loss of any single AZ.

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name ha-web-asg \
  --launch-template LaunchTemplateName=web-lt,Version=1 \
  --min-size 3 \
  --max-size 9 \
  --desired-capacity 3 \
  --vpc-zone-identifier "subnet-1a,subnet-1b,subnet-1c"

Enable Multi-AZ on RDS

Enables Multi-AZ on an existing RDS instance, creating a synchronous standby replica in a second AZ for automatic failover.

aws rds modify-db-instance \
  --db-instance-identifier my-db \
  --multi-az \
  --apply-immediately

Test AZ Failure with Instance Stop

Finds an ASG instance in a specific AZ and stops it to verify the Auto Scaling group replaces it automatically.

# Identify instances in a specific AZ
aws ec2 describe-instances \
  --filters Name=availability-zone,Values=us-east-1a \
            Name=tag:aws:autoscaling:groupName,Values=ha-web-asg \
  --query 'Reservations[].Instances[].InstanceId' \
  --output text | tr '\t' '\n' | head -1

# Stop one instance to simulate AZ failure; ASG will replace it
aws ec2 stop-instances --instance-ids i-0abc123

Discussion

Be the first to comment on this lesson.