What it Checks
For every URL and every bot you select, the tester does two things. It reads the site's robots.txt and tests the URL against the bot's token (GPTBot, ClaudeBot, PerplexityBot…), with the same longest-match rules as the robots.txt Tester. Then it requests the page with that bot's user agent and records the status, the size of the response, and whether the server sent content, refused the request or served a bot challenge instead.
A bot can access a URL only when robots.txt allows it and the request returns real content. The results are summed up per bot and per URL.
Run a Test
- Paste the URLs to check, one per line. Duplicates are removed. The counter shows how many fit with the bots selected.
- Keep the AI crawlers selected, or change the list of user agents.
- Press Test. The result opens with the verdict, for example All 35 bots can read the page. or 4 of 35 bots are kept out.
Options
- User agents. All AI crawlers and AI assistants' fetchers are selected by default: OpenAI, Anthropic, Perplexity, Google-Extended, Applebot-Extended, Common Crawl, Meta, ByteDance and others. You can add search engines or browsers from the user-agent list to compare their access with the AI bots'.
- Request pages as each bot. On by default: every URL is requested with every bot's user agent. A test then runs at most 1,000 requests (URLs × user agents) and 100 URLs: with all 35 AI bots selected that is 28 URLs, and the counter shows the limit for your selection. Turn it off to check robots.txt only: no page requests, up to 500 URLs with any number of bots, but no status codes or bot challenges.
- Sites. robots.txt is fetched once per site with a neutral user agent and once as each bot, so a test covers at most 1,000 robots.txt requests (sites × (bots + 1)): about 27 sites with all AI bots.
Reading the Result
- Verdict: how many bots can read every URL tested, with the robots.txt file of each site, the bots robots.txt blocks, the bots refused at request, and the bots the server answers a different robots.txt.
- Totals: checks run, checks with access, checks blocked by robots.txt, and checks blocked otherwise.
- By bot: for each bot, the company that runs it and how many of the URLs it may read.
- Results: one row per URL and bot, with the verdict, the robots.txt result, the deciding rule (for example
Disallow: / (line 12, group GPTBot)), the HTTP status, the response size and a note when robots.txt was refused to the bot or differs for it. The table can be filtered and exported.
What the verdict of a row means:
| Verdict | Meaning |
|---|---|
| Can access | robots.txt allows the bot and the page returned content. |
| Allowed by robots.txt | robots.txt allows the bot; the page wasn't requested (robots.txt-only test, or a token such as Google-Extended that never fetches). |
| Blocked by robots.txt | A rule disallows the URL for this bot, or robots.txt answered 429, 5xx or not at all (the whole site counts as disallowed). |
| Refused (HTTP 401/403) | robots.txt allows the bot, but the server refused the request. |
| Challenge page served | The server answered 2xx with a bot-protection page (Cloudflare, Imperva, DataDome…) instead of the content. |
| Rate limited, Empty response, HTTP …, Error | The request returned 429, fewer than 200 bytes, another status, or failed. |
Good to Know
- Some tokens only control use, not crawling.
Google-ExtendedandApplebot-Extendedare never sent as user agents by real crawlers; Googlebot and Applebot fetch the page and the token decides whether it may be used for AI training. These tokens (andWebzio-Extended) are checked against robots.txt and not requested. - robots.txt is fetched as each bot, because servers can answer bots differently. When a bot's copy is refused (401, 403, 429, 5xx or no answer) while the neutral request gets the file, the rules come from that file and the row says so: sites that verify crawlers by IP refuse impersonators, and the real bot most likely gets the file.
- Unknown user-agent ids and private network addresses are refused before anything is fetched.
- How groups and rules are matched is explained in the robots.txt guide.
API
POST /api/ai-bot-access
Send the URLs as one string, one per line, and optionally a list of user-agent ids (from the user-agent list). Without agents, all AI crawlers are tested. "requests": false checks robots.txt only. The answer is JSON; with "stream": true it is NDJSON instead: {"type": "progress", "done", "total"…} lines, then one {"type": "done", …} line with the result.
curl -X POST https://plumb.reacthive.dev/api/ai-bot-access \
-H "content-type: application/json" \
-d '{
"urls": "https://www.nytimes.com/\nhttps://www.nytimes.com/section/world",
"agents": ["ai-gptbot", "ai-claudebot"],
"requests": true
}'{
"rows": [
{
"url": "https://www.nytimes.com/",
"bot": "OpenAI",
"token": "GPTBot",
"robots": "blocked",
"id": "ai-gptbot",
"rule": "Disallow: / (line 233, group gptbot)",
"robotsStatus": 200,
"robotsNote": "",
"status": 200,
"bytes": 182344,
"finalUrl": "https://www.nytimes.com/",
"access": "Content served",
"verdict": "Blocked by robots.txt"
}
],
"urls": 2,
"requests": true,
"agents": ["OpenAI", "ClaudeBot"],
"robots": [{ "url": "https://www.nytimes.com/robots.txt", "status": 200 }]
}