robots.txt / robots.ts
Serve /robots.txt from a static file or a function returning crawl rules, sitemap and host.
export default function robots(): MetadataRoute.Robots- Added in Next.js 13.3.0
Parameters
| Parameter | Type | Description |
|---|---|---|
$rules | Rule | Rule[] | A single rule object, or an array of them to emit one User-Agent block per entry. |
$rules.userAgentoptional | string | string[] | Crawler name the block applies to, or an array of names. Use '*' for all crawlers. Optional on a single rule object, required on each element of an array. |
$rules.allowoptional | string | string[] | Path or paths the crawler may fetch. Emitted as one Allow line per value. |
$rules.disallowoptional | string | string[] | Path or paths the crawler must not fetch. Emitted as one Disallow line per value. |
$rules.crawlDelayoptional | number | Seconds a crawler should wait between requests. Not honoured by Googlebot. |
$rules.otheroptional | Record<string, string | number | Array<string | number>> | Non-standard directives such as Request-Rate or Clean-param, written verbatim inside this rule's User-Agent block. Keys keep their casing; array values emit one line each. |
$sitemapoptional | string | string[] | Absolute URL of a sitemap, or several. Emitted as Sitemap lines after the rule blocks. |
$hostoptional | string | Preferred host directive, recognised mainly by Yandex. |
Return value
MetadataRoute.RobotsA Robots object that Next.js serialises into Robots Exclusion Standard text and serves at /robots.txt.
Errors
Next.js does not validate directive names or values in `other`; a malformed directive is written out as given rather than raising an error.
Changelog
- PHP 16.3.0Added the per-rule `other` field for non-standard directives; values pass through unvalidated.
- PHP 13.3.0robots file convention introduced.
A plain app/robots.txt is served untouched. Swapping it for app/robots.ts makes it a Route Handler that returns a structured object, which Next.js formats into the Robots Exclusion Standard text.
When to reach for the code version
- Preview and staging deployments must disallow everything while production allows crawling — branch on
process.env.VERCEL_ENV. - The sitemap list is generated, so hard-coding
Sitemap:lines would drift. - You want different rules per crawler, for example blocking AI scrapers but not Googlebot.
What trips people up
Only one thing may own /robots.txt. A leftover public/robots.txt collides with app/robots.ts — delete the old file when you migrate, or you will be debugging why your edits have no effect.
The rules are a request to well-behaved crawlers, not access control: a disallowed URL can still be indexed if other sites link to it. Use a noindex tag for that. And when you pass an array of rules, userAgent becomes required on every element — a block without it silently applies to nobody useful.
Example
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: {
userAgent: '*',
allow: '/',
disallow: '/private/',
},
sitemap: 'https://acme.com/sitemap.xml',
}
}When to use it
- Block indexing of every preview deployment while leaving production crawlable
- Keep /admin and /api out of search results in one place
- Give Googlebot full access while disallowing Applebot and Bingbot
- Advertise several sharded sitemap URLs to crawlers
Frequently asked questions
Why is my robots.txt not updating in Next.js?
Usually a stale public/robots.txt is still being served — a static file in public/ occupies the same URL as app/robots.ts, so delete one. Failing that, the route is cached from build time; touch a request-time API or set a revalidate segment config to regenerate it.
Does Disallow in robots.txt stop a page appearing in Google?
No. Disallow only asks crawlers not to fetch the URL; Google can still index it from inbound links and show it without a snippet. To keep a page out of results, let it be crawled and return a noindex directive via the metadata robots field instead.
Can I use both robots.txt and robots.ts?
No — both claim /robots.txt, so keep exactly one in app/. Use the static .txt when the rules never change, and the .ts version when they depend on the environment or on a generated sitemap list.
More examples
Different rules per crawler
The array form emits one User-Agent block per rule, and an array of names expands into a separate block for each: Googlebot gets Allow / plus Disallow /private/, then Applebot and Bingbot each get their own block with Disallow /. The single Sitemap line is written once at the end, after all blocks.
// app/robots.ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
userAgent: 'Googlebot',
allow: ['/'],
disallow: '/private/',
},
{
userAgent: ['Applebot', 'Bingbot'],
disallow: ['/'],
},
],
sitemap: 'https://acme.com/sitemap.xml',
}
}Blocking crawlers on preview deployments
Preview builds serve a two-line file that disallows everything, so staging URLs never enter an index. Production emits Allow / with two Disallow lines, two Sitemap lines from the array, and a Host line. The environment check runs at build time, so each deployment bakes in the right answer.
// app/robots.ts
import type { MetadataRoute } from 'next'
const isProduction = process.env.VERCEL_ENV === 'production'
export default function robots(): MetadataRoute.Robots {
if (!isProduction) {
return {
rules: { userAgent: '*', disallow: '/' },
}
}
return {
rules: { userAgent: '*', allow: '/', disallow: ['/admin/', '/api/'] },
sitemap: [
'https://acme.com/sitemap.xml',
'https://acme.com/product/sitemap/0.xml',
],
host: 'https://acme.com',
}
}Non-standard per-agent directives
`other` writes Request-Rate: 10/1m inside the SeznamBot block only, with the key's casing preserved. Next.js does not check that the directive exists or that the value parses — it is copied through verbatim, so a typo ships silently. `crawlDelay` becomes a standard Crawl-delay: 10 line in the same block.
// app/robots.ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: '*', allow: '/' },
{
userAgent: 'SeznamBot',
allow: '/',
crawlDelay: 10,
other: {
'Request-Rate': '10/1m',
},
},
],
}
}
Discussion