What it Checks
The checker starts at the URL you enter and follows the links on the same host (www. or not), breadth first, up to the page limit. Every page it reads is parsed for links: <a href> and image map areas, <img src>, <script src>, <iframe src>, and the <link> tags that load something (stylesheets, icons, preloads, the manifest). Fragments are dropped, so /page#top and /page are one link.
Then each unique link is requested once: a HEAD request first, and a GET when HEAD isn't answered properly (some servers refuse or mishandle it). Redirects are followed hop by hop, up to 10. Pages the crawl already fetched aren't requested again. The result for each link is one of:
- Broken: the final answer is a 4xx or 5xx error (404, 410, 500…), a redirect loop, or a redirect without a target.
- No response or timeout: the host doesn't exist (DNS), refused the connection, or didn't answer within 12 seconds. A link with no response is tried a second time before it's reported.
- Redirect: the link works but goes through one or more redirects. Point it at the final URL.
- Refused: 401, 403, 429 or 999. The server turned the checker away (a login, a bot filter, a rate limit), which doesn't mean the link is dead. Open these in a browser.
Run a Test
- Enter the site's address (or any page of it). The crawl starts there.
- Press Check. A progress line shows the pages crawled and queued, then the links checked out of the total and the broken ones found so far. Cancel stops the run.
- The result opens with the verdict, for example 12 broken links on 4 pages. or No broken links.
Options
- Whole site or This page only. This page only reads the one page and checks the links on it, without crawling further.
- Max pages. How many pages to crawl, from 10 to 2,000 (100 by default). Links on pages beyond the limit aren't checked; links to those pages from crawled pages are.
- External links. On by default. Off checks only links to the same host.
- Images, scripts and CSS. On by default. Off checks only page links (
<a>and frames). - Respect robots.txt. On by default: pages the site's
robots.txtdisallows (for all crawlers,User-agent: *) aren't crawled, and internal links to them are listed as not checked. External links are always requested.
Reading the Result
- The verdict counts broken links, no responses and timeouts, and the pages they appear on. Refused links and redirects are not counted as broken.
- The tiles: unique links checked, broken (HTTP errors), redirects, timeouts and errors (no response), and pages crawled.
- The table lists one row per unique link: its status, the URL with what went wrong, the first page it's on (and how many more), its anchor text (or an image's alt text), internal or external, and its type. The tabs filter it: Broken, Redirects, Refused and All. Click a row for every page that uses the link (up to 25) and the full redirect chain.
- CSV and Excel export the rows of the current tab and filter.
| Status | Reported as |
|---|---|
| 2xx | OK, or Redirect when it took one or more hops |
| 401, 403, 429, 999 | Refused: check by hand |
| Other 4xx, 5xx | Broken |
| No answer | No response (DNS, connection) or Timeout |
Good to Know
- The checker is polite: 6 pages at a time while crawling, then at most 6 requests at a time to the site and 2 to each other host (16 in all). A large site takes a few minutes.
- Up to 5,000 unique links are checked per run. Links to private network addresses are never requested.
- Links added by JavaScript after the page loads aren't seen: the checker reads the HTML the server sends. The Fetch & Render tool shows what the rendered page contains.
- Redirects aren't errors, but each hop costs a request and can dilute signals. Update internal links to point at the final URL.
API
POST /api/broken-links
Send the URL and, optionally, scope ("site" or "page"), maxPages (1 to 2,000, default 500 through the API), maxDepth, external, resources and respectRobots (all true by default), and include / exclude path patterns (* matches anything). Invalid input answers HTTP 400 with { "error": "…" }.
The answer is a stream of JSON lines (application/x-ndjson): progress events while crawling (phase: "crawl") and checking (phase: "check"), then one done event with the result, or an error event.
curl -N -X POST https://plumb.reacthive.dev/api/broken-links \
-H "content-type: application/json" \
-d '{
"url": "http://www.deadlinkcity.com/",
"scope": "page",
"external": true,
"resources": false
}'{"type":"progress","phase":"crawl","crawled":1,"queued":0,"current":"http://www.deadlinkcity.com/"}
{"type":"progress","phase":"check","checked":24,"total":49,"broken":18,"current":"http://www.deadlinkcity.com/error-page.asp?e=410"}
{"type":"done","url":"http://www.deadlinkcity.com/","host":"www.deadlinkcity.com","scope":"page",
"robots":{"url":"http://www.deadlinkcity.com/robots.txt","status":200,"respected":true,"note":"robots.txt fetched; its rules apply."},
"stats":{"pages":1,"pagesWithBroken":1,"links":49,"occurrences":49,"broken":39,"errors":2,"timeouts":0,
"redirects":2,"blocked":3,"skipped":1,"ok":2,"internal":48,"external":1,"capped":false,"limitReached":false},
"pages":[{"url":"http://www.deadlinkcity.com/","status":200,"depth":0,"links":54}],
"links":[{"url":"http://www.deadlinkcity.com/default.asp?r=4","status":404,
"finalUrl":"http://www.deadlinkcity.com/nosuchpage2.txt",
"redirects":[{"url":"http://www.deadlinkcity.com/default.asp?r=4","status":301,"location":"http://www.deadlinkcity.com/nosuchpage2.txt"}],
"kind":"broken","note":"Not found","internal":true,"type":"page",
"pages":[{"url":"http://www.deadlinkcity.com/","text":"301 redirection to an invalid link","nofollow":false}],
"count":1,"method":"GET"}]}kind is one of broken, error, timeout, blocked (refused), redirect, skipped (disallowed by robots.txt or a private address) and ok; links are sorted with problems first. method says how the status was obtained: crawl, HEAD or GET.