Site Architecture & Crawlability

Design a flat, logical site structure that is easy to crawl.

Site architecture is how your pages are organized and linked. A good structure helps users and lets crawlers reach every important page.

Aim for a flat structure

Keep important pages within about three clicks of the homepage. Deeply buried pages get crawled less often and rank worse.

Crawlability essentials

Link to every important page from somewhere in the site.
Use a logical folder hierarchy (/blog/, /products/).
Provide an HTML navigation and an XML sitemap.
Watch your crawl budget on large sites - do not waste it on junk URLs.

Example

# A flat, logical hierarchy
home
  /coffee
    /coffee/pour-over-guide
    /coffee/best-gooseneck-kettle
  /tea
    /tea/green-tea-guide

When to use it

An SEO consultant flattens a deep e-commerce taxonomy from 6 levels to 3 so Googlebot can reach every product page within 3 clicks from the homepage.
A media site restructures its URL hierarchy into topic-based silos (/tech/, /finance/) to build topical authority clusters recognisable to crawlers.
A developer audits internal link depth reports and discovers 200 pages buried at click-depth 8, then adds them to relevant hub pages to improve discoverability.

More examples

Flat three-click site hierarchy

Keeping every page within three clicks of the homepage ensures Googlebot allocates crawl budget efficiently and distributes PageRank to all pages.

# Target: every page reachable in 3 clicks from homepage
https://example.com/                          # depth 0 — homepage
https://example.com/shoes/                    # depth 1 — category
https://example.com/shoes/trail-runners/      # depth 2 — subcategory
https://example.com/shoes/trail-runners/pro/  # depth 3 — product page

# Avoid: deep nesting beyond depth 4
# https://example.com/store/mens/footwear/sport/running/trail/pro-2024/

Topic silo HTML nav structure

Keeping cross-links within silos concentrates topical relevance on each section, helping Google understand the site's authority by subject area.

<nav aria-label="Main navigation">
  <ul>
    <!-- Each top-level item is a topical silo -->
    <li><a href="/technology/">Technology</a></li>
    <li><a href="/finance/">Finance</a></li>
    <li><a href="/health/">Health</a></li>
  </ul>
</nav>
<!-- Within /technology/ — only links to technology articles -->
<aside aria-label="Related articles">
  <a href="/technology/ai-trends-2024/">AI Trends 2024</a>
  <a href="/technology/best-laptops/">Best Laptops</a>
</aside>

Crawl depth report via Screaming Frog CLI

Generating a headless crawl depth report reveals which pages are buried too deep in the architecture and need additional internal links.

# Run Screaming Frog headless and export click-depth report
screaming_frog_seo_spider \
  --crawl https://example.com \
  --headless \
  --output-folder /tmp/sf-report \
  --export-tabs "Internal:All" \
  --save-crawl

# Summarise pages by click depth
awk -F',' 'NR>1 {print $10}' /tmp/sf-report/internal_all.csv \
  | sort | uniq -c | sort -rn

Discussion

Be the first to comment on this lesson.