robots.txt
Control which parts of your site crawlers may access with robots.txt.
Syntax
User-agent: *\nDisallow: /path/robots.txt is a plain-text file at the root of your domain that tells crawlers which URLs they may or may not request.
Key points
- It lives at
https://example.com/robots.txt. Disallowblocks crawling - but a blocked page can still be indexed if linked elsewhere.- To keep a page out of the index, use a
noindexmeta tag instead, and do not block it. - Reference your sitemap here so engines find it.
Example
# https://example.com/robots.txt
User-agent: *
Disallow: /cart/
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xmlWhen to use it
- A developer blocks the /admin/ and /checkout/ paths in robots.txt to prevent them from wasting Googlebot's crawl budget on non-indexable pages.
- An SEO audit discovers that a misconfigured robots.txt is blocking the entire site with 'Disallow: /', causing a complete disappearance from search results.
- A media company uses Disallow directives to exclude paginated URL parameters from crawling while keeping the base category pages fully accessible.
More examples
Standard production robots.txt
Blocking non-public paths preserves crawl budget for valuable pages while ensuring the sitemap is discoverable by all bots.
# /robots.txt — production site
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?q= # block internal search result pages
Allow: /admin/public-assets/ # but allow publicly needed assets
# Accelerate sitemap discovery
Sitemap: https://example.com/sitemap.xmlBot-specific crawl rate control
Crawl-delay limits third-party bots that generate server load, while Googlebot's rate is managed through Search Console to avoid penalties.
# Limit aggressive scrapers without affecting Googlebot
User-agent: AhrefsBot
Crawl-delay: 5
User-agent: SemrushBot
Crawl-delay: 10
# Never restrict Googlebot's crawl rate via robots.txt —
# use Search Console's crawl rate settings instead
User-agent: Googlebot
Allow: /Validate robots.txt via API
Fetching robots.txt and running Google's inspection endpoint catches accidental broad blocks before they cause indexation loss.
# Fetch and print robots.txt to check for accidental Disallow: /
curl -s https://example.com/robots.txt
# Check whether a specific URL would be blocked
# (Google's Robots Exclusion Protocol tester)
curl -s \
"https://searchconsole.googleapis.com/v1/sites/https%3A%2F%2Fexample.com/robotsTxt/inspect?url=https://example.com/blog/post-1&key=YOUR_KEY"
Discussion