Where it Lives
One file per host and protocol, at the root: https://www.example.com/robots.txt applies only to https://www.example.com, not to http://, example.com or shop.example.com. It must be plain text (UTF-8). Google reads the first 500 KiB.
What the HTTP Status Means
| Response | Google’s behaviour |
|---|---|
| 2xx | The rules apply. |
| 3xx | Followed for up to five redirects, then treated as 404. |
| 4xx (except 429) | No robots.txt: everything may be crawled. |
| 429, 5xx, network error | The whole site is treated as disallowed; after 30 days Google uses the last cached copy, or crawls freely if there’s none. |
Syntax
# Rules for every crawler
User-agent: *
Disallow: /cart/
Disallow: /*?sort=
Allow: /cart/help
# Googlebot gets its own group (and ignores the * group)
User-agent: Googlebot
Disallow: /drafts/
Sitemap: https://www.example.com/sitemap.xml- A group starts with one or more
User-agentlines followed byAllow/Disallowrules. Sitemaplines are independent of groups and need absolute URLs.- Field names are case-insensitive; path values are case-sensitive.
#starts a comment.
Which Group Applies
A crawler obeys exactly one group: the one whose user-agent matches its product token most specifically. Googlebot-Image uses a Googlebot-Image group if there is one, otherwise a Googlebot group, otherwise *. Groups naming the same crawler are merged. Rules from other groups are never combined, so a crawler with its own group ignores everything under User-agent: *, a common surprise when adding a group for one bot.
How Rules Match
- Rules match the path and query string from the start:
Disallow: /shopalso blocks/shoppingand/shop/item. *matches any sequence of characters;$anchors the end:Disallow: /*.pdf$blocks URLs ending in .pdf.- When several rules match, the one with the longest path wins. If an Allow and a Disallow are equally long, Allow wins.
- An empty
Disallow:allows everything;Disallow: /blocks the whole site.
| Rules | URL | Result |
|---|---|---|
Allow: /p, Disallow: / | /page | allowed (longer rule) |
Allow: /folder, Disallow: /folder | /folder/page | allowed (tie → allow) |
Allow: /page, Disallow: /*.htm | /page.htm | disallowed (longer rule) |
Disallow: /*? | /products?id=1 | disallowed |
Common Mistakes
- Using robots.txt to remove pages from Google. Blocked pages can stay indexed. To keep a page out, allow crawling and use
noindex(meta robots or X-Robots-Tag), which Google can only see if it may crawl the page. - Blocking CSS and JavaScript. Google renders pages; blocked resources can make pages look broken or not mobile-friendly.
- Unsupported directives. Google ignores
Crawl-delay,NoindexandNofollowin robots.txt (Bing and Yandex honour crawl-delay). - A staging file left in production.
Disallow: /copied from a test site blocks everything. - Server errors. A robots.txt that returns 5xx stops Google from crawling the whole site.
AI Crawlers
AI companies publish their own tokens: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Google-Extended (Gemini training; it doesn’t affect Search), Applebot-Extended and others. Some fetchers acting on a user’s request (ChatGPT-User, Perplexity-User) may not follow robots.txt the same way; check each company’s documentation, and use the AI Bot Access Tester to see what each one gets.