Skip to content
Plumb
Light theme onCheck a URL

DocsRendering

Fetch & Render

Fetch a page as a crawler would, render it in headless Chromium, and compare the raw HTML with the DOM after JavaScript.
  • Any user agent
  • JavaScript changes
  • Screenshot and requests
  • JSON API

What it Checks

The tool makes two requests with the user agent you choose. The first is a plain HTTP fetch, as a crawler's first pass: status, redirect chain, response headers and the raw HTML. The second loads the page in Chromium, runs its JavaScript and keeps the resulting DOM, a screenshot, every network request and the console output.

Both versions are read for the same SEO signals (title, meta description, robots directives, canonical, hreflang, headings, links, images, structured data, word count) and shown side by side. The text and links of both versions are also compared block by block, so you see exactly what JavaScript adds or removes: what crawlers that don't render (most AI and social bots) miss.

When robots.txt keeps scripts, stylesheets or data requests from loading, the page is rendered a second time with them allowed, and the two renderings are compared: you see whether the blocking changes what Google renders.

Run a Test

  1. Paste the URL of the page.
  2. Pick the user agent. Googlebot Smartphone is the default; mobile user agents render with a phone viewport.
  3. Press Fetch. The result opens with the verdict: Fetched and rendered., Rendered with 2 JavaScript errors., Redirected with 301., The server answered 404. or Blocked by robots.txt. When robots.txt itself can't be fetched (network error, 429 or 5xx), the verdict is robots.txt could not be fetched.: Google doesn't crawl the site until it can.
Open Fetch & Render

Options

  • User agent. Any crawler, AI bot, social fetcher or browser from the user-agent list. It is sent with both requests and decides which robots.txt group applies.
  • Render JavaScript. On by default. Turn it off to get only the raw fetch, which is faster.
  • Obey robots.txt. On by default. A disallowed page is not fetched at all, and while rendering, every resource is tested against the robots.txt of its own host; disallowed ones are not loaded, as a crawler would skip them.
  • Post-load wait. Seconds to wait after the page loaded and the network went quiet, for content that appears late (0 to 20, default 3).
  • Accept-Language and Region. Sends an Accept-Language header, for sites that adapt content to the language. Where a proxy is set up for a country, the region also sends the requests from that country.
  • Username and Password. HTTP basic authentication, for staging sites.

Reading the Result

  • Status, Redirects: the final status and each hop of the redirect chain, or the final URL when there was none.
  • Download, Render: the raw HTML's size and time, and the render's time with its number of requests and how many were blocked.
  • JavaScript: how much of the text is the same before and after JavaScript, and the words and links it adds or removes. See what changes opens the JavaScript changes tab.
  • Blocked resources (when robots.txt blocked scripts, stylesheets or data requests): whether the page renders the same without the blocking.
  • Signals: the SEO signals of the raw HTML next to the rendered DOM.
  • JavaScript changes: the text blocks and links only in the rendered DOM, and those JavaScript removed; and, with blocked resources, the requests blocked and the text that appears only when they are allowed.
  • Screenshot, Rendered HTML, Raw HTML, Headers: the page as Chromium drew it, the DOM after JavaScript, the response body before it, and the response headers.
  • Resources: every request the page made, with its type and status, marked blocked when robots.txt or a private address stopped it.
  • Console: uncaught JavaScript errors and console messages.

What a difference between the two columns usually means:

Raw HTML vs renderedMeaning
SameThe server sends the content; crawlers that don't render see it too.
Missing in raw, present renderedJavaScript adds it. Google sees it after rendering, later than the first crawl; most other crawlers and AI bots don't.
Different valuesJavaScript changes it, for example a title or canonical rewritten on load. Google may use either; keep them the same.

Good to Know

  • The screenshot covers the top 6,000 pixels of long pages; the Screenshot tab says when a page is longer.
  • Text is compared block by block (paragraphs, headings, list items, cells). The share in common counts three-word sequences, so a changed date or counter barely moves it. Links are compared without fragments, trailing slashes and tracking parameters such as utm_*.
  • Requests come from this server. A site that blocks data-centre traffic or verifies crawlers by IP can answer differently than it would to the real bot.
  • To compare what several user agents get without rendering, use the Pre-rendering Tester. For the canonical rules, see the rel=canonical guide; for robots.txt, the robots.txt guide.

API

POST /api/fetch-render

Send the URL and, optionally, the user-agent id, render and obeyRobots (both default to true), waitSeconds (0 to 20, default 3, as in the form), acceptLanguage, region, and username / password. When robots.txt blocks the page, the response holds only url, agent and blockedByRobots (the rule or the reason) and robotsStatus (the robots.txt status, null when it could not be fetched). rendered is null when rendering is off; its screenshot is a JPEG data URL, and screenshotClipped is true when the page is longer than 6,000px. jsDiff compares the raw HTML with the rendered DOM (lists stop at 150 text blocks and 300 links). blockedImpact is set only when robots.txt blocked scripts, stylesheets or data requests: the requests (blocked), the text in common between both renderings (textSimilarity), words and links with and without blocking (words, wordsUnblocked, links, linksUnblocked), the title without blocking and the text that only appears then (textOnlyUnblocked).

Request
curl -X POST https://plumb.reacthive.dev/api/fetch-render \
  -H "content-type: application/json" \
  -d '{
    "url": "https://example.com/",
    "agent": "googlebot-smartphone",
    "render": true,
    "obeyRobots": true,
    "waitSeconds": 3
  }'
Response (Trimmed)
{
  "url": "https://example.com/",
  "agent": "Googlebot Smartphone",
  "userAgent": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) ... (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
  "proxied": false,
  "raw": {
    "status": 200,
    "statusText": "OK",
    "finalUrl": "https://example.com/",
    "redirects": [],
    "headers": [["content-type", "text/html"]],
    "body": "<!doctype html><html lang=en>...",
    "bytes": 577,
    "timeMs": 13,
    "contentType": "text/html",
    "signals": { "title": "Example Domain", "wordCount": 25, "...": "..." }
  },
  "rendered": {
    "status": 200,
    "finalUrl": "https://example.com/",
    "html": "<!DOCTYPE html><html lang="en">...",
    "screenshot": "data:image/jpeg;base64,/9j/4AAQ...",
    "screenshotClipped": false,
    "requests": [
      { "url": "https://example.com/s.js", "type": "script", "method": "GET", "status": 200, "blocked": false }
    ],
    "console": [],
    "errors": [],
    "timeMs": 3800,
    "signals": { "title": "Example Domain", "wordCount": 127, "...": "..." }
  },
  "jsDiff": {
    "textSimilarity": 0.167,
    "wordsRaw": 25,
    "wordsRendered": 140,
    "textOnlyRendered": ["L'usage de ce domaine est réservé à des exemples de documentation, ...", "Learn more"],
    "textOnlyRaw": [],
    "linksOnlyRendered": ["https://iana.org/help/example-domains"],
    "linksOnlyRaw": []
  },
  "blockedImpact": null
}