What it Checks
The tester fetches the site's robots.txt and runs your URL through it the way Google does: it picks the group for the crawler you chose, finds every rule that matches the path and query, and lets the longest match win (an Allow beats a Disallow of the same length). You get the verdict, the exact rule and line that decided it, and the group that applied.
It also reads the file itself: syntax errors, misspelled directives Google forgives, directives it ignores (such as crawl-delay), and the sitemaps the file lists.
Run a Test
- Paste the full URL you want to check, including the path and query string.
- Pick the crawler. Googlebot Smartphone is the default because it is the one Google indexes with.
- Press Test. The result opens with the verdict: Allowed. or Blocked.
To test many URLs at once, open the Bulk tab: paste up to 1,000 URLs or paths (one per line; paths such as /cart resolve against the first full URL in the list), pick one or more crawlers, and press Test. Each URL is tested against its own site's robots.txt (up to 20 sites), or against the edited file in Editor mode.
Options
- Crawler. Search engines, AI crawlers, social fetchers and browsers. Each sends its own user agent and is matched by its robots.txt token (
Googlebot,GPTBot…). Google's specialised crawlers, such as Googlebot-Image, fall back to the Googlebot group when the file has none of their own. All tests the catch-allUser-agent: *group. - Live robots.txt or Editor. Live fetches the file from the site. Editor tests the rules you paste against the same URL, so you can try a change before publishing it. After a live test, Edit and retest copies the live file into the editor. With Compare with the live file (on by default) every verdict is also worked out with the live file, and verdicts your edit changes are marked (“now blocked”, “now allowed”).
- Compare major crawlers. Tests the URL for 32 crawlers at once: Google's crawlers and the Google-Extended token, Bing, DuckDuckGo, Yandex, Baidu, Apple, the main AI crawlers, social fetchers, Ahrefs and Semrush, each with its verdict, rule and group.
- Check resources. Renders the page in Chromium as the chosen crawler and tests every CSS, JavaScript, image and font request against the robots.txt of the host it comes from (up to 300 requests). Blocked resources can stop Google from rendering the page correctly.
Reading the Result
- Deciding rule: the line number and rule that won, for example
line 3 · Disallow: /search. When no rule matches, the URL is allowed. - Group: the
User-agentline whose rules applied. The file view highlights that group and the deciding rule. - Issues: problems in the file, by line. Errors are lines Google can't use; warnings are lines it reads differently than you may expect.
- Page resources (with Check resources): each request, its type, and the rule that blocks it, with totals for blocked requests and hosts.
- Bulk: one row per URL and crawler with the verdict and rule (and the live verdict and change in Editor mode), filters for blocked rows and changed verdicts, totals, the live files read and their status, and CSV or Excel export.
How the HTTP status of robots.txt itself changes the verdict:
| robots.txt answers | Result |
|---|---|
| 2xx | Its rules apply. |
| 4xx (except 429) | No file: everything is allowed. |
| 429, 5xx or no answer | Google treats the whole site as disallowed until it can read the file. |
Good to Know
- One file applies to one host and protocol:
https://www.example.com/robots.txtdoes not covershop.example.com. - robots.txt is fetched with the chosen crawler's user agent (in Bulk, the first one selected). Sites that verify crawlers by IP address refuse impersonators: when the crawler's request is refused (401, 403, 429, 5xx or no answer) but a neutral user agent gets the file, that file is used and a note says so, since the real crawler most likely gets it.
- Unknown crawler ids are refused with an error (the API never falls back to another crawler).
- The full rules are in the robots.txt guide.
API
POST /api/robots-txt
Send the URL and, optionally, the crawler id (from the user-agent list), the mode, the pasted file for the editor mode (robots; compare: false skips the live file), crawlers: "major" for the crawler comparison, and whether to check resources. For bulk, send urls (one per line, or a list) and agents (ids) instead of url and agent.
curl -X POST https://plumb.reacthive.dev/api/robots-txt \
-H "content-type: application/json" \
-d '{
"url": "https://www.google.com/search?q=plumb",
"agent": "googlebot-smartphone",
"mode": "live",
"resources": false
}'{
"url": "https://www.google.com/search?q=plumb",
"mode": "live",
"agent": { "id": "googlebot-smartphone", "name": "Googlebot Smartphone", "token": "Googlebot" },
"verdict": {
"allowed": false,
"rule": { "type": "disallow", "value": "/search", "line": 3 },
"agent": "*",
"groupLines": [1, 2],
"note": ""
},
"robots": {
"url": "https://www.google.com/robots.txt",
"status": 200,
"issues": [],
"sitemaps": ["https://www.google.com/sitemap.xml"],
"groups": 4
},
"live": null,
"crawlers": null,
"resources": null
}curl -X POST https://plumb.reacthive.dev/api/robots-txt -H "content-type: application/json" -d '{
"urls": "https://www.google.com/
/search?q=1
/maps",
"agents": ["googlebot", "ai-gptbot"],
"mode": "live"
}'{
"bulk": true,
"mode": "live",
"urls": 3,
"agents": [{ "id": "googlebot", "name": "Googlebot", "token": "Googlebot" }, …],
"fetchedAs": "Googlebot",
"files": [{ "url": "https://www.google.com/robots.txt", "status": 200, "issues": 0, "sitemaps": 1, "groups": 4 }],
"rows": [
{
"url": "https://www.google.com/search?q=1",
"id": "googlebot",
"name": "Googlebot",
"allowed": false,
"rule": "Disallow: /search",
"line": 3,
"matched": "*",
"note": ""
}
]
}In Editor mode with the comparison on, each row (and each crawler row) also has live (the live file's verdict) and changed.