Guides

Short, sourced answers on making a site readable to AI crawlers.

• AI crawlers and robots.txt: every user agent and what it controls

  The user agents OpenAI, Anthropic, Perplexity, Google and Apple use, what blocking each one does, and copy-paste robots.txt rules.

• llms.txt: what it is, who reads it, and a template

  What the llms.txt proposal is, what the 2026 adoption data says about whether AI systems request it, and a template you can copy.

• JavaScript and AI crawlers: why single-page apps look empty

  Why most AI crawlers see a blank page on JavaScript-rendered sites, how to tell if yours is affected, and the fixes for React, Vue and other frameworks.

• How the CrawlReadable score works

  Every check, its weight, and what an outside check can't see.