Skip to content
Plumb
Light theme onCheck a URL

DocsGuides

robots.txt

robots.txt tells crawlers which URLs of a site they may request. It controls crawling, not indexing: a blocked URL can still be indexed (without its content) when other pages link to it.
  • Guide

Where it Lives

One file per host and protocol, at the root: https://www.example.com/robots.txt applies only to https://www.example.com, not to http://, example.com or shop.example.com. It must be plain text (UTF-8). Google reads the first 500 KiB.

What the HTTP Status Means

ResponseGoogle’s behaviour
2xxThe rules apply.
3xxFollowed for up to five redirects, then treated as 404.
4xx (except 429)No robots.txt: everything may be crawled.
429, 5xx, network errorThe whole site is treated as disallowed; after 30 days Google uses the last cached copy, or crawls freely if there’s none.

Syntax

# Rules for every crawler
User-agent: *
Disallow: /cart/
Disallow: /*?sort=
Allow: /cart/help

# Googlebot gets its own group (and ignores the * group)
User-agent: Googlebot
Disallow: /drafts/

Sitemap: https://www.example.com/sitemap.xml
  • A group starts with one or more User-agent lines followed by Allow / Disallow rules.
  • Sitemap lines are independent of groups and need absolute URLs.
  • Field names are case-insensitive; path values are case-sensitive.
  • # starts a comment.

Which Group Applies

A crawler obeys exactly one group: the one whose user-agent matches its product token most specifically. Googlebot-Image uses a Googlebot-Image group if there is one, otherwise a Googlebot group, otherwise *. Groups naming the same crawler are merged. Rules from other groups are never combined, so a crawler with its own group ignores everything under User-agent: *, a common surprise when adding a group for one bot.

How Rules Match

  • Rules match the path and query string from the start: Disallow: /shop also blocks /shopping and /shop/item.
  • * matches any sequence of characters; $ anchors the end: Disallow: /*.pdf$ blocks URLs ending in .pdf.
  • When several rules match, the one with the longest path wins. If an Allow and a Disallow are equally long, Allow wins.
  • An empty Disallow: allows everything; Disallow: / blocks the whole site.
RulesURLResult
Allow: /p, Disallow: //pageallowed (longer rule)
Allow: /folder, Disallow: /folder/folder/pageallowed (tie → allow)
Allow: /page, Disallow: /*.htm/page.htmdisallowed (longer rule)
Disallow: /*?/products?id=1disallowed

Common Mistakes

  • Using robots.txt to remove pages from Google. Blocked pages can stay indexed. To keep a page out, allow crawling and use noindex (meta robots or X-Robots-Tag), which Google can only see if it may crawl the page.
  • Blocking CSS and JavaScript. Google renders pages; blocked resources can make pages look broken or not mobile-friendly.
  • Unsupported directives. Google ignores Crawl-delay, Noindex and Nofollow in robots.txt (Bing and Yandex honour crawl-delay).
  • A staging file left in production. Disallow: / copied from a test site blocks everything.
  • Server errors. A robots.txt that returns 5xx stops Google from crawling the whole site.

AI Crawlers

AI companies publish their own tokens: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), Google-Extended (Gemini training; it doesn’t affect Search), Applebot-Extended and others. Some fetchers acting on a user’s request (ChatGPT-User, Perplexity-User) may not follow robots.txt the same way; check each company’s documentation, and use the AI Bot Access Tester to see what each one gets.