Skip to content
Plumb
Light theme onCheck a URL

robots.txt Tester

Check whether a URL is blocked, which rule blocks it and for which crawler, and whether the page's CSS, JavaScript and images can be crawled. How it Works

Try

1

The Verdict

Allowed or blocked for the crawler you pick, or for 32 major crawlers at once, with the exact rule and group.

2

The File

Syntax errors, misspellings Google forgives, unsupported directives, and the sitemaps it lists. Edit it and see which verdicts change against the live file.

3

In Bulk

Up to 1,000 URLs or paths against each site's robots.txt or your edited file, with CSV export.

A Real Example

Checked 17 November 2026

https://www.google.com/search/about/

What the tool found
Googlebot has no group of its own in google.com's robots.txt, so it follows the * group. Disallow: /search (line 3) and Allow: /search/about (line 4) both match; the Allow is the longer path, so the URL is allowed.
What to do
Nothing to fix: this is how to keep a section out of crawling while opening part of it. The longer path wins whatever order the lines are in.
Run This Example

Reading the Result

  • Allowed

    No rule blocks the URL for that crawler, or an Allow rule wins. The page may be crawled; whether it's indexed is a separate question.

  • Blocked

    A Disallow rule matches. Google won't fetch the page, but it can still index the bare URL if other pages link to it. To keep a page out of results, let it be crawled and add noindex.

  • Which Rule Wins

    The most specific user-agent group applies, then the longest matching path; on a tie, Allow beats Disallow.

  • No File or an Error

    A 404 means everything may be crawled. A 5xx makes Google stop crawling the site until robots.txt answers again.

What it Can't Tell You

  • It shows what the rules say, not what Google has crawled. Search Console's robots.txt report shows the copy Google last fetched.
  • robots.txt is a request, not access control: crawlers that ignore it aren't stopped by it.