Skip to content
Plumb
Light theme onCheck a URL

DocsGuides

llms.txt

llms.txt is a markdown file at /llms.txt that gives AI agents a short, curated map of a site: what it is, and links to the pages worth reading, ideally in markdown. It is a proposal (llmstxt.org), not a web standard, and it has nothing to do with search rankings.
  • Guide

What it Is for

Agents read websites constantly: a coding assistant fetches a library’s documentation to get an API call right, a chat assistant reads a product’s pages to answer a question. HTML pages wrap their content in navigation, scripts and layout, and a whole site doesn’t fit in a context window. llms.txt gives an agent one small file to start from: a summary of the project and a list of links, with notes, to the detailed content. The agent reads or searches the file, then fetches only the pages it needs.

The proposal was published by Jeremy Howard (Answer.AI) in September 2024. Version 2 (August 2026) added link relations for discovery, allowed both page.md and page.html.md for markdown versions, defined what a file in a subfolder covers, and dropped the special meaning of the Optional section.

Where it Lives

At /llms.txt, or at any path: /docs/llms.txt covers everything under /docs/. When several files apply, agents use the most specific one. Serve it as text/plain or text/markdown, in UTF-8.

The Format

The file is markdown with a fixed order, so it can be parsed with plain code as well as read by a model:

  1. An optional byte-order mark.
  2. An H1 with the name of the project or site. The only required part.
  3. A blockquote with a short summary: what the project is, and what an agent needs to understand the rest.
  4. Details: any markdown except headings (paragraphs, lists), such as important notes on how to use the project.
  5. H2 sections, each a “file list”: a markdown list where each item is a link [name](url), optionally followed by : and notes about the file.
# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX,
> and fastcore's FT "FastTags" into a library for server-rendered hypermedia apps.

Important notes:

- It is not compatible with FastAPI syntax.
- It works with vanilla JS libraries, but not with React, Vue or Svelte.

## Docs

- [Quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): All HTMX attributes, events and config options

## Examples

- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): A complete CRUD app

## Optional

- [Starlette documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/starlette-sml.md): A subset useful for FastHTML

By convention a section named Optional holds secondary links an agent can skip when it needs a shorter context. Since v2 it is a convention only, with no mechanical meaning.

Rules Parsers Rely on

  • Exactly one H1, first in the file. Further H1s, or text before it, break the structure.
  • No headings before the first H2: the details are any markdown except headings.
  • In sections, each list item starts with the link: - [Name](https://…): notes. A link in the middle of a sentence, notes after a dash instead of a colon, or a section with prose and no links are read differently by parsers (the reference parser takes only list items starting with a link).
  • Absolute URLs: the file is often read away from its address, so relative links lose their base.
  • Small enough to fit in a context window. The detail belongs behind the links. Some published files run to 100 KB and more; that is a sign content should move into linked pages.

Markdown Versions of Pages

The links should point to content agents read easily. The proposal suggests a clean markdown version of each page at the same URL with .md appended (page.html.md) or the extension replaced (page.md); URLs without a file name use index.html.md or index.md. Documentation platforms such as Mintlify, GitBook and nbdev generate these automatically.

To let agents find them, v2 recommends standard link relations, as HTML or as an HTTP header (which also works for the markdown files themselves):

<link rel="alternate" type="text/markdown" href="/docs/page.html.md">
<link rel="describedby" href="/docs/llms.txt">

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

llms-full.txt

Many sites also publish /llms-full.txt: the whole documentation concatenated into one markdown file, so a tool can load everything in one request. It is not part of the proposal, but a widespread convention (Mintlify and other doc platforms generate it, and sites such as Svelte and Anthropic publish one). These files are large, often megabytes, so they suit search and retrieval more than a context window. Some sites add intermediate sizes, such as Svelte’s llms-medium.txt and llms-small.txt. Keep llms.txt itself the short index.

How it Differs from robots.txt and Sitemaps

robots.txtsitemap.xmlllms.txt
PurposeWhich URLs crawlers may fetchEvery indexable URL, for search enginesA curated guide for agents answering a question
FormatPlain-text directivesXMLMarkdown
ContentRules, no descriptionsURLs with datesSummary, notes, links (external ones too), ideally to markdown
UsedBefore crawlingFor discovery and indexingOn demand, when an agent needs information (inference, rarely training)
StatusStandard (RFC 9309)Protocol supported by all major enginesA proposal

llms.txt doesn’t grant or deny access: an AI crawler blocked in robots.txt stays blocked whatever llms.txt says. And a sitemap is no substitute: it lists everything, usually without markdown versions, and far more than fits in a context window.

Adoption

  • Thousands of sites publish one, mostly developer documentation: Anthropic, OpenAI, Google’s Gemini API docs, Stripe, Vercel, Cloudflare, Svelte and React among them.
  • Documentation platforms and CMSs generate it: Mintlify, GitBook, Wix, the Yoast SEO and AIOSEO WordPress plugins, plugins for VitePress, Docusaurus and Drupal.
  • Chrome’s Lighthouse checks for one in its agentic browsing audits; coding agents and MCP servers fetch them to load documentation.
  • Search engines don’t use it. Google has said it doesn’t read llms.txt for Search, and no AI company has committed to reading it on every request. Treat it as a cheap, useful aid for agents, not a ranking factor.

Writing a Good One

  • Write the summary for a reader who knows nothing about the project: what it is, who it is for, what is unusual about it.
  • Put the warnings an agent would otherwise get wrong in the details (deprecated APIs, incompatibilities, version notes).
  • List the pages that answer real questions, with notes that say what each contains. Leave out navigation, legal pages and archives.
  • Link markdown versions where they exist.
  • Test it: give an agent only your llms.txt and ask it questions about your product.