llms.txt: what it is, what it does not do, and how to write one

What the llms.txt proposal specifies

Jeremy Howard published the proposal on 3 September 2024. The current text at llmstxt.org is version 2, modified on 10 August 2026. It is an informal overview kept in a public GitHub repository and open for community input, not an IETF or W3C standard. No search engine or AI provider is obliged to read it.

The file is Markdown, with its parts in a fixed order:

1. An H1 with the site or project name. This is the only required part.
2. A blockquote with a short summary of the key facts.
3. Optional paragraphs or lists with more context, without headings.
4. Zero or more H2 sections with file lists: list items that hold a link, optionally followed by a colon and a note, such as - [Pricing](https://example.com/pricing): plans and limits.

A section named Optional holds secondary links that an agent can skip when it needs a shorter context.

The file can sit at the root or under a path such as /docs/llms.txt. It covers the URLs under its path, and when several files apply, agents should use the most specific one. The proposal does not use /.well-known/, because well-known locations exist only at the origin root and many authors control only a path on a shared host.

The proposal also suggests clean Markdown copies of pages at the same URL, with .md appended (page.html.md) or, since version 2, with the extension replaced (page.md). Version 2 adds two link relations, sent as HTML <link> elements or an HTTP Link header: rel="alternate" type="text/markdown" for the copy and rel="describedby" for the llms.txt that covers the page. The current text does not define a companion file such as llms-full.txt; when a tool or documentation platform produces one, that is its own convention.

How llms.txt differs from robots.txt and sitemap.xml

The three files do different jobs for different readers.

The proposal draws the same line: robots.txt says what automated access is acceptable, while llms.txt is used on demand. It adds that a sitemap is no substitute, because it often lacks Markdown versions of pages, does not include URLs on other sites and usually covers more content than fits in a model's context window.

So if you want to allow or block an AI crawler, edit robots.txt. Leaving a page out of llms.txt does not hide it, and listing a page there does not override a robots.txt rule that blocks a crawler. The robots.txt guide explains how rules are matched.

What Google and AI providers have said about llms.txt

Here is what the primary sources say:

• Google Search. Its guide to generative AI features (updated 10 July 2026) says you do not need machine-readable files, AI text files or Markdown to appear in Google Search, including its AI features, because Search does not use them. It names llms.txt directly: keeping one for other services that use such files is fine, but it neither helps nor harms visibility or rankings in Google Search.
• Chrome Lighthouse. The experimental Agentic Browsing category (Chrome 150 or later, no 0–100 score) has an llms.txt audit that checks for the file at the domain root. A server error is flagged; a 404 is marked not applicable, because providing the file is optional. That is a presence check in a developer tool, not a statement that any search or AI product uses the file.
• OpenAI. Its crawler page explains robots.txt controls for OAI-SearchBot (search results in ChatGPT) and GPTBot (training), and notes that user-initiated ChatGPT-User visits may not follow robots.txt. It does not say that OpenAI reads a site's llms.txt.
• Anthropic. Its help article lists ClaudeBot, Claude-User and Claude-SearchBot and says they honor robots.txt. It does not mention llms.txt.
• Perplexity. Its documentation covers PerplexityBot for search results and the user-triggered Perplexity-User fetcher, which generally ignores robots.txt. It does not say that Perplexity reads a site's llms.txt.
• Microsoft Bing. No Microsoft or Bing guidance on llms.txt was found. The February 2026 announcement of AI Performance in Bing Webmaster Tools does not mention the file.

As of 1 October 2026, none of these providers documents using a site's llms.txt for ranking, citations or answers. Several of them publish one for their own developer documentation (OpenAI, Anthropic, Perplexity and Google's Gemini API docs), as does Cloudflare. Publishing a file for your own docs is not the same as reading other sites' files.

When llms.txt is worth adding, and when it can wait

The file is cheap to write and, according to Google, harmless in Search. The real cost is keeping it correct.

A quick test: would a developer paste your URL into a coding assistant and ask it to work with your product? If yes, an llms.txt with Markdown copies of the key pages is an afternoon well spent. If you want to be named in answers about a local service, spend that afternoon on clear page content and crawler access instead. If agents should act on your product rather than read about it, look at an API or an MCP server; the MCP introduction explains how that works.

A minimal llms.txt example for a small site

A complete file for a hypothetical scheduling app at example.com:

# Example Rota

> Example Rota is a browser-based app for scheduling shifts in small cafés and shops. A free plan covers one location; paid plans add more locations.

Example Rota does not run payroll. Last reviewed: 2026-10-01.

## Docs

- [Getting started](https://example.com/docs/getting-started.md): Create a location, add staff and publish a first week
- [Shift templates](https://example.com/docs/templates.md): Repeating shifts, breaks and cover rules
- [API reference](https://example.com/docs/api.md): REST endpoints, authentication and rate limits

## Product

- [Pricing](https://example.com/pricing): Plans, limits and what each plan includes
- [Data and privacy](https://example.com/privacy): Where data is stored and how to export or delete it

## Optional

- [Changelog](https://example.com/changelog): Release notes by month

• Serve it as text. https://example.com/llms.txt should return status 200 and the Markdown itself. Single-page apps sometimes answer unknown paths with their HTML shell, so check the response with curl.
• Markdown copies are optional. If you do not publish them, link the normal HTML pages, as the Product section above does.
• Keep the summary factual. State what the product is, who it is for and its important limits. Leave slogans out.
• Write notes, not prompts. Asking a model to recommend you adds nothing a reader can check.
• Be selective. Ten good links beat two hundred; the full list of URLs belongs in the sitemap.

With Markdown copies, each HTML page can point to its copy and to the file that covers it:

<link rel="alternate" type="text/markdown" href="/docs/getting-started.md">
<link rel="describedby" href="/llms.txt">

Generators, checkers and validators: keeping the file accurate

A file that lists an old price or a deleted page can be worse than none, because an agent that follows it may repeat outdated facts.

• Build it with your docs. If the documentation is generated from Markdown, produce llms.txt and the page copies in the same build step.
• Put it on the release checklist. When prices, plan names, platforms or URLs change, update the file in the same release.
• Check the links automatically. Every URL should answer 200 without a chain of redirects. This shell one-liner prints the status code of every absolute link in the file:

curl -s https://example.com/llms.txt | grep -oE 'https://[^) ]+' | while read -r u; do echo "$(curl -s -o /dev/null -w '%{http_code}' "$u") $u"; done

An llms.txt generator is fine for a first draft built from a sitemap or a docs folder. It cannot decide which pages matter, and it may include pages set to noindex or blocked in robots.txt. Review the output like any page you publish.

An llms.txt checker or validator can confirm the mechanical parts: the file is reachable, the H1 comes first and the links resolve. No checker can tell you whether an assistant reads the file, because the providers do not document that.

What matters more for AI answers

If you want to be a usable source for AI answers, these come first:

1. Crawl access for the bots you want. AI crawlers are controlled in robots.txt, each under its own token. OpenAI separates OAI-SearchBot (ChatGPT search) from GPTBot (training), Anthropic separates Claude-SearchBot from ClaudeBot, and Perplexity documents PerplexityBot. The Google-Extended token is a robots.txt control, not a separate crawler: it governs whether content Google crawls may be used for Gemini training and grounding and, according to Google, does not affect inclusion or ranking in Google Search. Allowing search crawlers while blocking training crawlers is a legitimate choice. Keep in mind that a User-agent: * group with Disallow: / blocks every crawler that follows robots.txt and has no group of its own. OpenAI says a robots.txt change can take about 24 hours to reach its systems.
2. Authentication for private content. User-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt, and RFC 9309 says robots rules are not access authorization.
3. Clear content on the page. Google's guidance for its AI features points to unique, useful content and a clear technical structure that is ready for discovery and indexing, not to special files. Answer real questions directly in the HTML of a reachable page. The AI visibility guide works through an example.
4. Structured data that matches the page. Google says structured data is not required for its generative AI search and that no special schema.org markup is needed; it still recommends structured data for rich-result eligibility in Search. If you use JSON-LD, keep it consistent with the visible facts.
5. Consistent facts. Name, prices, plans, regions and contact details should agree across your pages, your structured data and your llms.txt. If your own pages disagree, any system summarizing them has to guess.

No file can make an assistant cite you. What you control is whether your pages are reachable, readable and correct; the technical SEO guide covers the crawl and indexing basics.

What Sitelemetry checks about llms.txt

Sitelemetry's AI visibility area is part of the six-area audit (security, technical SEO, AI visibility, accessibility, performance and integrations), which starts at $49/month on the Starter plan. Its llms.txt check is narrow:

• It requests /llms.txt once at the origin and reports the status and size.
• The file counts as present only if it returns a 2xx status, is not empty, is not HTML and looks like a link guide: a line starting with # or -, a URL, or words such as docs or sitemap. A missing file is reported as a partial result with a recommendation.
• It does not validate the format, does not follow or test the links, and does not claim that any AI system reads the file.

The same area checks robots.txt for site-wide blocks against 16 named AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. Rules under User-agent: * and partial disallows are not attributed to individual AI crawlers (a site-wide block for all crawlers is reported separately), and the list is fixed, so check crawlers that are not on it, such as Claude-SearchBot or Claude-User, yourself. It also reports JSON-LD structured data without validating it against schema.org. The audit reads static HTML from a sample of pages without running JavaScript, and it does not query ChatGPT, Claude, Perplexity or Google or track citations in AI answers.

The Free plan covers security checks only. The free Launch Readiness Snapshot runs eight passive checks without signup; it does not look at llms.txt, robots.txt or structured data, and it is not the six-area audit. Plan details are on the pricing page.