robots.txt Tester
Check whether a URL is blocked, which rule blocks it and for which crawler, and whether the page's CSS, JavaScript and images can be crawled. How it Works
The Verdict
Allowed or blocked for the crawler you pick, or for 32 major crawlers at once, with the exact rule and group.
The File
Syntax errors, misspellings Google forgives, unsupported directives, and the sitemaps it lists. Edit it and see which verdicts change against the live file.
In Bulk
Up to 1,000 URLs or paths against each site's robots.txt or your edited file, with CSV export.
A Real Example
Checked 17 November 2026https://www.google.com/search/about/
- What the tool found
- Googlebot has no group of its own in google.com's robots.txt, so it follows the * group. Disallow: /search (line 3) and Allow: /search/about (line 4) both match; the Allow is the longer path, so the URL is allowed.
- What to do
- Nothing to fix: this is how to keep a section out of crawling while opening part of it. The longer path wins whatever order the lines are in.
Reading the Result
Allowed
No rule blocks the URL for that crawler, or an Allow rule wins. The page may be crawled; whether it's indexed is a separate question.
Blocked
A Disallow rule matches. Google won't fetch the page, but it can still index the bare URL if other pages link to it. To keep a page out of results, let it be crawled and add noindex.
Which Rule Wins
The most specific user-agent group applies, then the longest matching path; on a tie, Allow beats Disallow.
No File or an Error
A 404 means everything may be crawled. A 5xx makes Google stop crawling the site until robots.txt answers again.
What it Can't Tell You
- It shows what the rules say, not what Google has crawled. Search Console's robots.txt report shows the copy Google last fetched.
- robots.txt is a request, not access control: crawlers that ignore it aren't stopped by it.