Drift Detection & Troubleshooting

Detect out-of-band changes with plan/refresh, then read plans carefully and reach for TF_LOG when things get weird.

Drift is when the real world stops matching your state — someone clicked in the console, an autoscaler resized things, a manual hotfix went in. Undetected drift is how "it works on my machine" becomes "the apply deleted the fix."

Detecting drift

  • terraform plan — a clean plan (no changes) means no drift. Run it on a schedule in CI to catch surprises early.
  • terraform plan -refresh-only — shows how state differs from reality without proposing config changes.

Reading a plan like a pro

  • + create, - destroy, ~ update in place.
  • -/+ means replace — the scary one. Always find out why before approving.

When you are truly stuck

Set TF_LOG=DEBUG to see the raw provider API calls. It is verbose, but it is how you find out which exact request is failing and why.

Example

Example · bash
# Is reality still matching state? A clean plan = no drift.
terraform plan -detailed-exitcode
#   0 = no changes, 1 = error, 2 = drift/changes present

# Show ONLY how the world drifted from state
terraform plan -refresh-only

# Reconcile state with reality (no config changes)
terraform apply -refresh-only

# Something baffling? Turn on the firehose.
TF_LOG=DEBUG terraform plan 2> debug.log

When to use it

  • An operator runs terraform plan on a schedule every hour to detect out-of-band changes made in the AWS console and alert the team before they cause bigger problems.
  • An SRE uses TF_LOG=DEBUG when a plan fails with an opaque provider error, finding the root cause in the detailed API request and response logs.
  • A team adds a drift-detection job to their CI pipeline that runs terraform plan -detailed-exitcode and pages on-call if exit code 2 (changes detected) is returned.

More examples

Detect drift with plan

Uses -refresh-only to see only state drift (no config changes) and -detailed-exitcode to signal drift to a calling script.

Example · bash
# Run plan against live infrastructure
terraform plan -refresh-only
# If output shows changes, someone modified infra outside Terraform

# Exit code 2 means drift was found
terraform plan -detailed-exitcode
echo "Exit code: $?"  # 0=no changes, 1=error, 2=drift found

Enable verbose logging

Enables DEBUG logging to a file so the full API request/response cycle can be inspected when a plan or apply produces cryptic errors.

Example · bash
# Set log level: TRACE, DEBUG, INFO, WARN, ERROR
export TF_LOG=DEBUG
export TF_LOG_PATH=/tmp/terraform-debug.log

terraform apply

# Inspect the log
grep 'Error\|error\|WARN' /tmp/terraform-debug.log | head -50

Scheduled drift detection in CI

Runs a scheduled GitHub Actions job every hour that fails and alerts if Terraform detects out-of-band infrastructure drift.

Example · yaml
name: Drift Detection
on:
  schedule:
    - cron: '0 * * * *'  # every hour

jobs:
  detect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - name: Check for drift
        run: |
          set +e
          terraform plan -detailed-exitcode -refresh-only
          if [ $? -eq 2 ]; then
            echo "DRIFT DETECTED" && exit 1
          fi

Related

tf plan apply destroytf state commands

Discussion

  • Be the first to comment on this lesson.