What it Checks
Crawl a site (the default) starts at the URL you enter and follows links on the same host (www. or not), breadth first, up to the page limit. Each page is fetched once, redirects followed. A URL goes in the sitemap only when it is a page search engines should index:
- it answers 200 directly (URLs that redirect are left out; their target is listed instead, once);
- it is HTML (links to PDFs, images and other files aren't pages);
- it has no
noindexin its meta robots orX-Robots-Tagheader, and its canonical, when it has one, points to itself (both on by default, see options); robots.txtallows it (with Respect robots.txt).
Every other URL the crawl met is listed with the reason it was left out.
From a CSV builds sitemaps with hreflang annotations: one row per page, one column per language version, the header row holding the hreflang codes (en-US, fr, x-default…). Every URL of a row gets its own <url> entry listing all versions of the row as xhtml:link alternates, itself included, as Google requires. Language codes and URLs are checked as the file is built.
Either way, above 50,000 URLs the output is split into several sitemaps plus a sitemap index.
Run a Test
- On Crawl a site, enter the site's home page and press Generate. A progress line shows the pages crawled, queued and skipped, and the URL being fetched; Cancel stops the crawl.
- The result opens with the count, for example 124 URLs in the sitemap., then the file and the table of every URL found.
- Copy the XML, or download it (XML) or the plain list of URLs (URL list), and publish it on your site.
- For hreflang sitemaps, open From a CSV (hreflang), paste the CSV or choose a file, and press Create sitemap. Download the template gives a starting file.
Options
Crawling:
- Max pages. From 10 to 2,000 pages (500 by default). URLs found beyond the limit are listed as not crawled.
- Respect robots.txt. On by default: URLs disallowed for all crawlers (
User-agent: *) aren't fetched; they're listed as left out. - Also start from the current sitemap. Adds the URLs of
/sitemap.xmland of the sitemapsrobots.txtdeclares (indexes followed) to the crawl, so pages no link reaches are found too.
Output (changing these updates the file at once, without crawling again):
- Change frequency. None (the default), a fixed value (
always…never), or Auto:dailyfor the start page,weeklyone click away,monthlydeeper. - Priority. None (the default), the same value for all pages, or Auto: 1.00 for the start page, then ×0.8 per click of depth (0.80, 0.64, 0.51…), never below 0.10.
- Last modified. From the server (the default) uses each page's
Last-Modifiedheader and omits the tag when there is none; Now uses the crawl time; or none. - Include images. Adds the images each page shows as
image:imageentries (up to 1,000 per page). - Leave out noindex and non-canonical pages. On by default. Off lists them too.
From a CSV: lastmod, changefreq and priority columns are copied into every entry of the row; a url (or loc) column makes plain entries without alternates. The output is named after the uploaded file.
Reading the Result
- The verdict gives the URLs in the sitemap, the pages crawled and the time, and the URLs left out grouped by reason.
- The tiles: URLs found, pages crawled, in the sitemap, left out.
- The file, ready to copy or download; one tab per file when the output is split.
- Pages: every URL, whether it's in the sitemap, its HTTP status and title, its depth (clicks from the start page) and why it's left out. Filter by In the sitemap or Left out, and export to CSV or Excel.
| Reason | Meaning |
|---|---|
| Redirects | The URL answers 3xx; its target is listed instead. |
| Error status / No response | 4xx or 5xx, or no answer (DNS, timeout). |
| noindex | The page asks not to be indexed. |
| Canonical elsewhere | Its rel=canonical names another URL; that URL belongs in the sitemap. |
| Not HTML | A file (PDF, image…) rather than a page. |
| robots.txt / Page limit | Not crawled: disallowed, or beyond Max pages. |
Good to Know
- The crawler reads the HTML the server sends. Links that only appear after JavaScript runs aren't followed; use Also start from the current sitemap or the CSV mode for such sites.
- Below the generator, Notify search engines of changes submits changed URLs to IndexNow (Bing, Yandex, Seznam, Naver, Yep). After a crawl, one click fills it with the sitemap's URLs. Your key file must be published on the site first.
- Requests come from this server, 6 at a time, and never to private network addresses. For hreflang sitemaps, check the result with the hreflang Tags Tester once it's online; the rules are in the hreflang guide.
API
POST /api/sitemap-crawl
Send the URL and, optionally, maxPages (1 to 2,000, default 500), maxDepth, respectRobots (default true), sitemaps (start from the current sitemap), include / exclude path patterns (* matches anything), and the output options changefreq (none, auto, always … never), priority (none, auto, fixed with priorityValue), lastmod (none, server, now), images, excludeNoindex and excludeNonCanonical. Invalid input answers HTTP 400 with { "error": "…" }.
The answer is a stream of JSON lines (application/x-ndjson): progress events while crawling, then one done event with every URL found (rows, each with included and reason) and the sitemap files, or an error event.
curl -N -X POST https://plumb.reacthive.dev/api/sitemap-crawl \
-H "content-type: application/json" \
-d '{
"url": "https://www.scrapethissite.com/",
"maxPages": 20,
"priority": "auto",
"lastmod": "server"
}'{"type":"progress","crawled":1,"queued":2,"current":"https://www.scrapethissite.com/","skipped":2}
{"type":"progress","crawled":10,"queued":5,"current":"https://www.scrapethissite.com/pages/frames/?frame=i&family=Chelydridae","skipped":32}
{"type":"done","url":"https://www.scrapethissite.com/","host":"www.scrapethissite.com","origin":"https://www.scrapethissite.com",
"crawledAt":"2026-10-21T21:11:22.485Z",
"robots":{"url":"https://www.scrapethissite.com/robots.txt","status":200,"respected":true,"note":"robots.txt fetched; its rules apply."},
"stats":{"crawled":20,"found":52,"included":3,"excluded":{"noindex":15,"status":2,"robots":2,"limit":30},
"skippedNotListed":0,"sitemapUrls":0,"limitReached":true,"timeMs":2191},
"rows":[
{"url":"https://www.scrapethissite.com/","status":200,"depth":0,"title":"Scrape This Site | A public sandbox for learning web scraping",
"html":true,"canonical":"","noindex":false,"lastmod":"","redirectsTo":"","included":true,"code":"","reason":""},
{"url":"https://www.scrapethissite.com/pages/frames/","status":200,"depth":2,"noindex":true,
"included":false,"code":"noindex","reason":"noindex (meta robots or X-Robots-Tag)"}],
"files":[{"name":"sitemap.xml","urls":3,
"xml":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n <url>\n <loc>https://www.scrapethissite.com/</loc>\n <priority>1.00</priority>\n </url>…"}]}POST /api/indexnow
The CSV mode runs in your browser and has no API. The IndexNow submission does: send your key, the changed URLs as one string (one per line, all on the same host, up to 10,000) and, optionally, the key file's location. The key file must answer 200 with exactly the key; it defaults to https://<host>/<key>.txt.
curl -X POST https://plumb.reacthive.dev/api/indexnow \
-H "content-type: application/json" \
-d '{
"key": "3f9c1d7a5b2e4f60a8c9d1e2f3a4b5c6",
"urls": "https://example.com/fr/\nhttps://example.com/fr/a-propos/"
}'{
"ok": true,
"status": 202,
"keyLocation": "https://example.com/3f9c1d7a5b2e4f60a8c9d1e2f3a4b5c6.txt",
"count": 2,
"message": "Accepted: the key will be validated, then the URLs processed."
}