Crawling, Indexing & Ranking
Follow a page through the crawl to index to rank pipeline step by step.
Let's trace the full journey a page takes from being published to appearing in search results.
Crawl
Google finds your URL through links, sitemaps or previously known pages, then fetches the HTML.
Render & index
Google renders the page (running JavaScript if needed), extracts the content and meaning, and adds it to the index. You can request indexing in Search Console, but you cannot force it.
Rank
For each query, hundreds of signals decide the order: relevance, quality, links, page experience and more.
Where pages get stuck
- Blocked in robots.txt - never crawled.
- noindex tag - crawled but kept out of the index.
- Thin or duplicate content - indexed but ranks poorly.
Example
<!-- This meta tag tells engines NOT to index a page -->
<meta name="robots" content="noindex, follow">
<!-- The default (indexable) behaviour needs no tag, but can be stated explicitly -->
<meta name="robots" content="index, follow">When to use it
- A content team submits a new article URL via the Indexing API to speed up crawling and avoid waiting weeks for natural discovery.
- An SEO audit flags pages blocked by noindex directives that should be ranking, allowing the team to fix them before a product launch.
- An e-commerce site adds internal links to newly created category pages to raise their crawl frequency and get them indexed quickly.
More examples
Request immediate indexing via API
The Indexing API signals Google to crawl a URL immediately rather than waiting for the next scheduled crawl cycle.
# Google Indexing API — notify Google of a new or updated URL
curl -X POST \
'https://indexing.googleapis.com/v3/urlNotifications:publish' \
-H 'Authorization: Bearer YOUR_ACCESS_TOKEN' \
-H 'Content-Type: application/json' \
-d '{"url": "https://example.com/blog/new-post", "type": "URL_UPDATED"}'Block a page from the index
The noindex directive tells crawlers to exclude the page from the index, keeping internal or duplicate pages out of search results.
<!-- Prevent a staging or duplicate page from appearing in search -->
<head>
<meta name="robots" content="noindex, nofollow">
</head>
<!-- Equivalent via HTTP response header -->
<!-- X-Robots-Tag: noindex, nofollow -->Inspect crawl path via server logs
Parsing server logs reveals exactly which pages Googlebot crawled, how often, and which returned errors during the crawl phase.
# Extract Googlebot visits from nginx access log
grep -i 'googlebot' /var/log/nginx/access.log \
| awk '{print $7, $9}' \
| sort | uniq -c | sort -rn \
| head -20
# Output: count /path status_code
Discussion