Skip to content
Plumb
Light theme onCheck a URL

DocsCrawling

Sitemap Generator

Crawl a site and get its XML sitemap, with the pages left out and why; or build sitemaps with hreflang alternates from a CSV. Then tell search engines which URLs changed.
  • Crawls up to 2,000 pages
  • hreflang from a CSV
  • Streaming JSON API

What it Checks

Crawl a site (the default) starts at the URL you enter and follows links on the same host (www. or not), breadth first, up to the page limit. Each page is fetched once, redirects followed. A URL goes in the sitemap only when it is a page search engines should index:

  • it answers 200 directly (URLs that redirect are left out; their target is listed instead, once);
  • it is HTML (links to PDFs, images and other files aren't pages);
  • it has no noindex in its meta robots or X-Robots-Tag header, and its canonical, when it has one, points to itself (both on by default, see options);
  • robots.txt allows it (with Respect robots.txt).

Every other URL the crawl met is listed with the reason it was left out.

From a CSV builds sitemaps with hreflang annotations: one row per page, one column per language version, the header row holding the hreflang codes (en-US, fr, x-default…). Every URL of a row gets its own <url> entry listing all versions of the row as xhtml:link alternates, itself included, as Google requires. Language codes and URLs are checked as the file is built.

Either way, above 50,000 URLs the output is split into several sitemaps plus a sitemap index.

Run a Test

  1. On Crawl a site, enter the site's home page and press Generate. A progress line shows the pages crawled, queued and skipped, and the URL being fetched; Cancel stops the crawl.
  2. The result opens with the count, for example 124 URLs in the sitemap., then the file and the table of every URL found.
  3. Copy the XML, or download it (XML) or the plain list of URLs (URL list), and publish it on your site.
  4. For hreflang sitemaps, open From a CSV (hreflang), paste the CSV or choose a file, and press Create sitemap. Download the template gives a starting file.
Open the Sitemap Generator

Options

Crawling:

  • Max pages. From 10 to 2,000 pages (500 by default). URLs found beyond the limit are listed as not crawled.
  • Respect robots.txt. On by default: URLs disallowed for all crawlers (User-agent: *) aren't fetched; they're listed as left out.
  • Also start from the current sitemap. Adds the URLs of /sitemap.xml and of the sitemaps robots.txt declares (indexes followed) to the crawl, so pages no link reaches are found too.

Output (changing these updates the file at once, without crawling again):

  • Change frequency. None (the default), a fixed value (always … never), or Auto: daily for the start page, weekly one click away, monthly deeper.
  • Priority. None (the default), the same value for all pages, or Auto: 1.00 for the start page, then ×0.8 per click of depth (0.80, 0.64, 0.51…), never below 0.10.
  • Last modified. From the server (the default) uses each page's Last-Modified header and omits the tag when there is none; Now uses the crawl time; or none.
  • Include images. Adds the images each page shows as image:image entries (up to 1,000 per page).
  • Leave out noindex and non-canonical pages. On by default. Off lists them too.

From a CSV: lastmod, changefreq and priority columns are copied into every entry of the row; a url (or loc) column makes plain entries without alternates. The output is named after the uploaded file.

Reading the Result

  • The verdict gives the URLs in the sitemap, the pages crawled and the time, and the URLs left out grouped by reason.
  • The tiles: URLs found, pages crawled, in the sitemap, left out.
  • The file, ready to copy or download; one tab per file when the output is split.
  • Pages: every URL, whether it's in the sitemap, its HTTP status and title, its depth (clicks from the start page) and why it's left out. Filter by In the sitemap or Left out, and export to CSV or Excel.
ReasonMeaning
RedirectsThe URL answers 3xx; its target is listed instead.
Error status / No response4xx or 5xx, or no answer (DNS, timeout).
noindexThe page asks not to be indexed.
Canonical elsewhereIts rel=canonical names another URL; that URL belongs in the sitemap.
Not HTMLA file (PDF, image…) rather than a page.
robots.txt / Page limitNot crawled: disallowed, or beyond Max pages.

Good to Know

  • The crawler reads the HTML the server sends. Links that only appear after JavaScript runs aren't followed; use Also start from the current sitemap or the CSV mode for such sites.
  • Below the generator, Notify search engines of changes submits changed URLs to IndexNow (Bing, Yandex, Seznam, Naver, Yep). After a crawl, one click fills it with the sitemap's URLs. Your key file must be published on the site first.
  • Requests come from this server, 6 at a time, and never to private network addresses. For hreflang sitemaps, check the result with the hreflang Tags Tester once it's online; the rules are in the hreflang guide.

API

POST /api/sitemap-crawl

Send the URL and, optionally, maxPages (1 to 2,000, default 500), maxDepth, respectRobots (default true), sitemaps (start from the current sitemap), include / exclude path patterns (* matches anything), and the output options changefreq (none, auto, always … never), priority (none, auto, fixed with priorityValue), lastmod (none, server, now), images, excludeNoindex and excludeNonCanonical. Invalid input answers HTTP 400 with { "error": "…" }.

The answer is a stream of JSON lines (application/x-ndjson): progress events while crawling, then one done event with every URL found (rows, each with included and reason) and the sitemap files, or an error event.

Request
curl -N -X POST https://plumb.reacthive.dev/api/sitemap-crawl \
  -H "content-type: application/json" \
  -d '{
    "url": "https://www.scrapethissite.com/",
    "maxPages": 20,
    "priority": "auto",
    "lastmod": "server"
  }'
Response (NDJSON, Trimmed)
{"type":"progress","crawled":1,"queued":2,"current":"https://www.scrapethissite.com/","skipped":2}
{"type":"progress","crawled":10,"queued":5,"current":"https://www.scrapethissite.com/pages/frames/?frame=i&family=Chelydridae","skipped":32}
{"type":"done","url":"https://www.scrapethissite.com/","host":"www.scrapethissite.com","origin":"https://www.scrapethissite.com",
 "crawledAt":"2026-10-21T21:11:22.485Z",
 "robots":{"url":"https://www.scrapethissite.com/robots.txt","status":200,"respected":true,"note":"robots.txt fetched; its rules apply."},
 "stats":{"crawled":20,"found":52,"included":3,"excluded":{"noindex":15,"status":2,"robots":2,"limit":30},
          "skippedNotListed":0,"sitemapUrls":0,"limitReached":true,"timeMs":2191},
 "rows":[
  {"url":"https://www.scrapethissite.com/","status":200,"depth":0,"title":"Scrape This Site | A public sandbox for learning web scraping",
   "html":true,"canonical":"","noindex":false,"lastmod":"","redirectsTo":"","included":true,"code":"","reason":""},
  {"url":"https://www.scrapethissite.com/pages/frames/","status":200,"depth":2,"noindex":true,
   "included":false,"code":"noindex","reason":"noindex (meta robots or X-Robots-Tag)"}],
 "files":[{"name":"sitemap.xml","urls":3,
   "xml":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n  <url>\n    <loc>https://www.scrapethissite.com/</loc>\n    <priority>1.00</priority>\n  </url>…"}]}

POST /api/indexnow

The CSV mode runs in your browser and has no API. The IndexNow submission does: send your key, the changed URLs as one string (one per line, all on the same host, up to 10,000) and, optionally, the key file's location. The key file must answer 200 with exactly the key; it defaults to https://<host>/<key>.txt.

Request
curl -X POST https://plumb.reacthive.dev/api/indexnow \
  -H "content-type: application/json" \
  -d '{
    "key": "3f9c1d7a5b2e4f60a8c9d1e2f3a4b5c6",
    "urls": "https://example.com/fr/\nhttps://example.com/fr/a-propos/"
  }'
Response
{
  "ok": true,
  "status": 202,
  "keyLocation": "https://example.com/3f9c1d7a5b2e4f60a8c9d1e2f3a4b5c6.txt",
  "count": 2,
  "message": "Accepted: the key will be validated, then the URLs processed."
}