Skip to content
Plumb
Light theme onCheck a URL

DocsCrawling

llms.txt Tool

Validate a site's llms.txt against the llmstxt.org format and check every link it lists, or generate a draft llms.txt from the site's sitemap and pages.
  • llmstxt.org v2 format
  • Link checks
  • Draft generator
  • JSON API

What it Checks

Validate finds the site's llms.txt and reads it the way llms.txt parsers do: one H1 with the project name, an optional blockquote summary, optional details (any markdown except headings), then H2 sections that are lists of links written - [name](url): notes. Every problem is reported by line, and every linked URL is fetched to see whether it answers and whether it serves markdown or plain text, which agents read best.

Generate builds a draft from the site itself: its name and summary from the home page, its pages from the sitemap (or the home page's links), each page's title and meta description as the link name and notes, grouped by path. Where the site serves a markdown version of a page, the draft links to it.

Run a Check

  1. On Validate, enter the site (https://example.com/), a path (https://example.com/docs/) or the file itself (https://example.com/llms.txt).
  2. Press Validate. The result opens with the verdict: Valid llms.txt., N problems. or No llms.txt.
  3. To build a file instead, switch to Generate, enter the site and press Generate. Edit the draft, then copy or download it; Validate runs the draft through the validator.
Open the llms.txt Tool

Options

  • Live file or Paste. Live fetches the file. A file covers the pages under its path and the most specific one applies, so for /docs/guide the tool tries /docs/guide/llms.txt, then each parent folder up to /llms.txt (four places at most) and uses the first that answers 200 with text. Paste validates text you enter; the optional base URL resolves relative links. After a live check, Edit and revalidate copies the file into the editor.
  • Check links. Fetches each distinct URL listed in the sections (up to 300, eight at a time, reading at most 64 KB of each). Turn it off for a quick format check.
  • Pages (Generate): how many pages to list, 25, 50 or 100. Pages are picked across sections, shallow pages first, so a large sitemap isn't represented by its first folder only.
  • Find .md versions (Generate): for each page, a rel="alternate" type="text/markdown" link, or the URL forms the proposal names (page.md, page.html.md, index.md). If the first three pages have none, the rest aren't tried.

Reading the Result

  • Verdict. Problems are errors, warnings and broken links. Notes (grey) are suggestions and don't count.
  • Facts. The file's URL, status and content type; title; summary; sections with their link counts (Optional in blue); size, lines and an estimate of tokens (bytes ÷ 4); whether an llms-full.txt sits beside it and its size; and whether the page you entered points to the file with rel="describedby".
  • Issues, by line. Issues that repeat inside a section are grouped into one, with the lines listed. The file view highlights the lines with errors, warnings and broken links.
  • Links: status, final URL after redirects, section, type (markdown or text are what the proposal asks for, html works but costs agents more), size and a note.
SeverityExamples
ErrorNo H1, more than one H1, content before the H1, malformed links, invalid URLs, an HTML page served at /llms.txt.
WarningSections without links, links not at the start of a list item, notes without a colon, relative URLs, headings in the details, a wrong content type, a file over 100 KB.
NoteNo summary, H3 subheadings in sections, prose or link-less items in a section, duplicate links, Optional not last, a redirect to another host.

A link counts as broken when it answers 4xx or 5xx or can't be fetched. 401 and 403 (logins, bot protection) and 429 (rate limits) are shown but not counted.

Good to Know

  • The generated file is a draft. A good llms.txt is curated: a summary written for agents, the pages that answer real questions, notes on each.
  • Generation reads one sitemap index and up to six of its child sitemaps (image, video and news sitemaps skipped), pages without a query string, and skips sign-in, cart, search and pagination URLs and pages marked noindex. Tag, category, author and legal pages go under Optional unless they are most of the site. Dated URLs (/2024/05/post/) form one Posts section. A page whose title only repeats its file name (ed for /ed.html) is named by its H1 instead.
  • Files over 1,500 lines show their first 1,500 lines; Show all renders the rest. Issues and highlights cover the whole file.
  • Requests come from this server. Sites behind bot protection may answer differently than they do to an agent; a refused link shows its status.

API

POST /api/llms-txt

Validate a live file (source: "live", the default) or pasted text (source: "paste" with content; url is then optional and only resolves relative links). checkLinks defaults to true. The answer is JSON; with "stream": true it is NDJSON: {"type": "progress", "phase": "links", "done", "total"} lines while links are checked, then one {"type": "done", …} line with the result.

Request
curl -X POST https://plumb.reacthive.dev/api/llms-txt \
  -H "content-type: application/json" \
  -d '{ "url": "https://llmstxt.org/", "source": "live", "checkLinks": true }'
Response (Trimmed)
{
  "source": "live",
  "url": "https://llmstxt.org/",
  "found": true,
  "file": {
    "url": "https://llmstxt.org/llms.txt",
    "finalUrl": "https://llmstxt.org/llms.txt",
    "status": 200,
    "contentType": "text/plain; charset=utf-8",
    "bytes": 637,
    "tried": [{ "url": "https://llmstxt.org/llms.txt", "status": 200, "note": "found" }]
  },
  "text": "# llms.txt\n\n> A proposal that those interested in ...",
  "parsed": {
    "title": "llms.txt",
    "summary": "A proposal that those interested in providing LLM-friendly content ...",
    "sections": [{ "name": "Docs", "line": 5, "optional": false, "links": 3 }],
    "links": 3,
    "lineCount": 10
  },
  "issues": [],
  "counts": { "errors": 0, "warnings": 0, "notices": 0, "broken": 0 },
  "highlight": [],
  "full": { "url": "https://llmstxt.org/llms-full.txt", "status": 404, "found": false, "bytes": null },
  "discovery": { "page": "https://llmstxt.org/", "describedby": null, "markdown": null },
  "links": [
    {
      "url": "https://llmstxt.org/index.md",
      "line": 7,
      "name": "llms.txt proposal",
      "section": "Docs",
      "status": 200,
      "contentType": "text/markdown",
      "bytes": 13606,
      "kind": "markdown",
      "ok": true,
      "note": ""
    }
  ],
  "unchecked": 0
}

Each issue is { "severity": "error" | "warning" | "notice", "line": 12, "lines": [12, 14], "message": "…" } (line is null for the whole file; lines only on grouped issues).

POST /api/llms-txt/generate

Generate a draft. maxPages is 5 to 100 (default 50); markdown (default true) looks for .md versions. A URL with a path limits the draft to the pages under it.

Request
curl -X POST https://plumb.reacthive.dev/api/llms-txt/generate \
  -H "content-type: application/json" \
  -d '{ "url": "https://llmstxt.org/", "maxPages": 50, "markdown": true }'
Response (Trimmed)
{
  "url": "https://llmstxt.org/",
  "scope": "https://llmstxt.org/",
  "site": { "name": "llms-txt", "summary": "A proposal to standardise on using an /llms.txt file ..." },
  "source": "sitemap",
  "sitemaps": ["https://llmstxt.org/sitemap.xml"],
  "found": 7,
  "text": "# llms-txt\n\n> A proposal to standardise ...\n\n## Pages\n\n- [Changes](https://llmstxt.org/changes.md)\n...",
  "pages": [
    {
      "url": "https://llmstxt.org/changes.html",
      "finalUrl": "https://llmstxt.org/changes.html",
      "status": 200,
      "title": "Changes",
      "description": "",
      "section": "Pages",
      "markdownUrl": "https://llmstxt.org/changes.md",
      "included": true,
      "reason": ""
    }
  ],
  "markdown": 7,
  "existing": "https://llmstxt.org/llms.txt"
}