Crawl Errors & Redirects

Diagnose crawl errors and use redirects correctly to protect rankings.

Broken pages and bad redirects waste crawl budget and lose ranking signals. Keep your site clean.

Common status codes

200 - OK, indexable.
301 - permanent redirect; passes ranking signals to the new URL.
302 - temporary redirect; use only when the move is temporary.
404 - not found; fine for truly removed pages.
410 - gone; a stronger "removed" signal.

Redirect tips

Use 301 when a URL changes permanently.
Avoid redirect chains (A to B to C) - point straight to the final URL.
Fix internal links that point to redirected or broken URLs.

Example

# Example nginx 301 redirect for a moved page
location = /old-coffee-guide {
    return 301 https://example.com/coffee/pour-over-guide;
}

When to use it

An SEO manager uses Google Search Console's Coverage report to identify 150 pages returning 404 errors and schedules a redirect mapping sprint to recover their link equity.
A developer sets up automated monitoring of server logs to alert the team whenever Googlebot encounters 5xx errors, allowing same-day fixes before rankings drop.
An e-commerce site cleans up a chain of 302 temporary redirects that should be 301s, restoring the full link equity pass-through to product pages.

More examples

301 redirect in nginx for moved content

301 redirects transfer ranking signals and link equity to the destination URL; using 302 (temporary) instead leaks authority and can confuse crawlers.

server {
    listen 443 ssl;
    server_name example.com;

    # Permanent redirect: pass full link equity to new URL
    location = /old-shoes-page/ {
        return 301 /shoes/trail-runner-pro/;
    }

    # Wildcard redirect for renamed category
    location /running/ {
        return 301 /shoes$request_uri;
    }
}

Find 4xx errors in access logs

Filtering server logs for Googlebot 404s reveals broken internal links and missing redirects that are wasting crawl budget and losing link equity.

# Extract URLs returning 404 that Googlebot hit in the last 7 days
grep -i 'googlebot' /var/log/nginx/access.log \
  | awk '$9 == 404 {print $7}' \
  | sort | uniq -c | sort -rn \
  | head -20
# Output: count  /broken-url

Check redirect chain depth

Tracing the full redirect chain from origin to final URL surfaces redirect loops or chains longer than 2 hops that dilute link equity.

python3 - << 'EOF'
import requests

def trace_redirects(url):
    r = requests.get(url, allow_redirects=True, timeout=8)
    chain = [resp.url for resp in r.history] + [r.url]
    print(f"Final URL: {r.url}  ({r.status_code})")
    print(f"Hops: {len(r.history)}")
    for i, hop in enumerate(r.history):
        print(f"  {i+1}. [{hop.status_code}] {hop.url}")

trace_redirects("https://example.com/old-shoes-page/")
EOF

Discussion

Be the first to comment on this lesson.