Skip to content
Plumb
Light theme onCheck a URL

AI Bot Access Tester

Check whether AI crawlers and assistants (GPTBot, ClaudeBot, PerplexityBot…) may crawl your pages: robots.txt, status code and content per bot, for up to 100 URLs (500 for robots.txt only). How it Works

0 / 28 URLs · 0 / 1,000 requests
Up to 28 URLs with 35 user agents. Off: robots.txt only, up to 500 URLs.

Try

1

robots.txt

Each AI crawler's verdict and rule, from robots.txt fetched as that bot.

2

The Real Request

A request with each bot's user agent: status, size, refusals and bot challenges. Or robots.txt only, for up to 500 URLs.

3

The Overview

Which bots can read your pages and which are kept out, per URL and per bot.

3 selected tokens (Google-Extended, Applebot-Extended, Webzio-Extended) only control use of content other bots fetch: they are checked against robots.txt, not requested.

A Real Example

Checked 17 November 2026

https://www.theguardian.com/

What the tool found
Of 35 AI user agents, 11 got the page and 15 are blocked by one shared Disallow: / on line 102, which names ClaudeBot, PerplexityBot, CCBot, Applebot-Extended and others. GPTBot, OAI-SearchBot and ChatGPT-User aren't named, so the * group lets them in. Another 7 that robots.txt allows, Perplexity-User and Claude-Web among them, were refused with HTTP 403.
What to do
A deliberate choice: the file lists the bots it keeps out and leaves OpenAI's unnamed. If you copy the pattern, keep robots.txt and your CDN's bot rules in step, so a bot isn't allowed by one and refused by the other.

Reading the Result

  • Can Access

    robots.txt allows the bot, and the page came back with content to a request carrying its user agent.

  • Blocked by robots.txt

    A Disallow rule covers the URL in the bot's own group, or in the * group when the file doesn't name it. The Rule column gives the line to change if that isn't what you meant.

  • Refused, Rate Limited or Challenge Page

    robots.txt allows the bot, but the server or CDN turned the request away with 401, 403 or 429, or served a bot check instead of the page. Look at your CDN's bot settings: they can block AI crawlers without any change to robots.txt.

  • Allowed by robots.txt

    Shown for Google-Extended, Applebot-Extended and Webzio-Extended, which never fetch pages: they decide whether content the company's other crawlers fetched may be used for its AI models. Also shown for every bot when pages aren't requested.

What it Can't Tell You

  • robots.txt is voluntary. The tool shows what the file asks of each bot, not whether the company honours it, or what it does with content it already has.
  • Real crawlers come from published IP ranges that bot filters trust; these requests don't. A 403 here can mean the filter caught an impersonator and would let the real bot in, so treat the request results as a lead and the robots.txt verdicts as exact.