CrawlReadable

llms.txt: what it is, who reads it, and a template

llms.txt is a Markdown file at the root of a site that summarises it for large language models: a title, a one-line description, and links to the pages that matter. It's a proposal from llmstxt.org, not a standard. In 2026 about one in ten sites has one, but server log studies found that 97% of those files receive no AI requests at all.

What goes in the file

The proposal at llmstxt.org asks for an H1 with the site name, a blockquote summary, then sections of links with one-line descriptions:

# Example Co

> Example Co makes scheduling software for dental clinics.

## Product

- [Features](https://example.com/features): what the software does
- [Pricing](https://example.com/pricing): plans and prices

## Docs

- [API reference](https://example.com/docs/api): REST API for bookings

Publish it at /llms.txt. A copy at /.well-known/llms.txt covers tools that look there. An optional /llms-full.txt carries the full text of your docs in one file.

Does anyone read it?

Adoption grew fast: the number of sites with llms.txt rose 8.8 times to May 2026 (Originality.ai, reported by ppc.land), and measured adoption ranges from 8.7% of the top 1,000 sites to 11.8% across 151.6 million live sites (Rankability, June 2026; Stackscan, 2026).

Usage didn't follow. A 2026 analysis reported by ppc.land found 97% of llms.txt files received zero requests from AI systems, and Stackscan found two fifths of files were auto-generated by Wix and GoDaddy.

So add one, because it takes an hour, but fix readable HTML and robots.txt first. That's why CrawlReadable gives llms.txt 4 points out of 100 and server-rendered content 40.

Common mistakes

Serving your HTML homepage at /llms.txt: many sites answer every unknown path with the homepage. We check that the file isn't HTML.

Listing every page: the point is a short, curated map. Link the 5 to 30 pages that explain what you do.

Letting it go stale: if your pricing page moved, the file now points AI systems at a 404.

Questions

Is llms.txt an official standard?

No. It's a community proposal published at llmstxt.org in 2024. No major AI company has committed to reading it.

Does llms.txt replace robots.txt or sitemap.xml?

No. robots.txt sets crawl rules and sitemap.xml lists URLs. llms.txt is an optional summary on top of both.

Check your own site

  • Fetching your page without JavaScript
  • Reading robots.txt for GPTBot, ClaudeBot, PerplexityBot
  • Measuring readable text and structure
  • Checking llms.txt and sitemap.xml
  • Scoring