AI Bot Access Tester
Check whether AI crawlers and assistants (GPTBot, ClaudeBot, PerplexityBot…) may crawl your pages: robots.txt, status code and content per bot, for up to 100 URLs (500 for robots.txt only). How it Works
robots.txt
Each AI crawler's verdict and rule, from robots.txt fetched as that bot.
The Real Request
A request with each bot's user agent: status, size, refusals and bot challenges. Or robots.txt only, for up to 500 URLs.
The Overview
Which bots can read your pages and which are kept out, per URL and per bot.
3 selected tokens (Google-Extended, Applebot-Extended, Webzio-Extended) only control use of content other bots fetch: they are checked against robots.txt, not requested.
A Real Example
Checked 17 November 2026https://www.theguardian.com/
- What the tool found
- Of 35 AI user agents, 11 got the page and 15 are blocked by one shared Disallow: / on line 102, which names ClaudeBot, PerplexityBot, CCBot, Applebot-Extended and others. GPTBot, OAI-SearchBot and ChatGPT-User aren't named, so the * group lets them in. Another 7 that robots.txt allows, Perplexity-User and Claude-Web among them, were refused with HTTP 403.
- What to do
- A deliberate choice: the file lists the bots it keeps out and leaves OpenAI's unnamed. If you copy the pattern, keep robots.txt and your CDN's bot rules in step, so a bot isn't allowed by one and refused by the other.
Reading the Result
Can Access
robots.txt allows the bot, and the page came back with content to a request carrying its user agent.
Blocked by robots.txt
A Disallow rule covers the URL in the bot's own group, or in the * group when the file doesn't name it. The Rule column gives the line to change if that isn't what you meant.
Refused, Rate Limited or Challenge Page
robots.txt allows the bot, but the server or CDN turned the request away with 401, 403 or 429, or served a bot check instead of the page. Look at your CDN's bot settings: they can block AI crawlers without any change to robots.txt.
Allowed by robots.txt
Shown for Google-Extended, Applebot-Extended and Webzio-Extended, which never fetch pages: they decide whether content the company's other crawlers fetched may be used for its AI models. Also shown for every bot when pages aren't requested.
What it Can't Tell You
- robots.txt is voluntary. The tool shows what the file asks of each bot, not whether the company honours it, or what it does with content it already has.
- Real crawlers come from published IP ranges that bot filters trust; these requests don't. A 403 here can mean the filter caught an impersonator and would let the real bot in, so treat the request results as a lead and the robots.txt verdicts as exact.