CrawlReadable

See what AI crawlers actually read on your website.

ChatGPT, Claude and Perplexity send crawlers that don't run JavaScript. They read the raw HTML your server sends, and plenty of sites send almost nothing. Get a free inspection, a score out of 100, and the fixes that matter.

  • Fetching your page without JavaScript
  • Reading robots.txt for GPTBot, ClaudeBot, PerplexityBot
  • Measuring readable text and structure
  • Checking llms.txt and sitemap.xml
  • Scoring

Free, no signup. We fetch your homepage, robots.txt, llms.txt and sitemap.xml. Takes about 10 seconds.

Certificate of AI readabilitySample
Site inspected
acme-analytics.com
Fetched as
GPTBot, no JavaScript
Readable text
14 characters
Checks failed
13 of 20

Points earned by category

  • Access14/26
  • Content8/40
  • Structure10/20
  • Discovery0/10
  • Interface0/4
A sample certificate for a site built with JavaScript. Every report gets a page like this, with a link you can share.
sites scored so far
291
average score across them
68/100
checks in every inspection
20
to deliver a fix pack
2 days

Same page, two readers

A site built with React, Vue or a no-code builder sends a nearly empty file and lets JavaScript draw the page in the browser. People never notice. AI crawlers do, because they read the file and stop there.

What a person sees in Chrome

What GPTBot receives

The same homepage twice. In a browser it shows a headline, a sign-up button and product tiles. Fetched the way GPTBot fetches it, the only readable text is the page title, Acme Analytics.

How well-known sites score

Each stamp links to the full report: what the crawler received, which bots are blocked, and every finding.

Just checked:

See every score on the leaderboard

Why it matters

An analysis of crawler traffic by Vercel and MERJ in December 2024 found that OpenAI's and Anthropic's crawlers download JavaScript files but never run them.

So a site can look finished to people and arrive empty at ChatGPT. Google does run JavaScript, which is how a site can rank well in Google and still be missing from AI answers.

A robots.txt rule copied from a template can do the same thing quietly: 5 of the first 17 well-known sites we checked block GPTBot or ClaudeBot.

Do AI crawlers run JavaScript?

How the score works

  1. We fetch your homepage like an AI crawler.One plain HTTP request, no browser, no JavaScript.
  2. We read what came back.robots.txt rules for each AI bot, the text in your HTML, your structured data, llms.txt and your sitemap.
  3. You get a score out of 100.Plus a certificate you can share, and fixes ranked by how many points each one is worth.

What counts, and how much

Readable content is 40 of the 100 points, because it's where most sites fail. llms.txt is only 4, because almost no AI system requests it yet.

Access, 26 points
Whether AI crawlers are allowed in: robots.txt rules for GPTBot, ClaudeBot and PerplexityBot, plus llms.txt.
Content, 40 points
Whether there's real text in the HTML before any JavaScript runs. The biggest category, because it's the most common failure.
Structure, 20 points
Whether the page describes itself: title, description, Open Graph, canonical link and JSON-LD.
Discovery, 10 points
Whether crawlers can find your other pages: a sitemap with recent dates.
Interface, 4 points
Whether AI agents can do more than read: an OpenAPI description or a full-text llms file.

Every check and its weight is on the methodology page.

Who's behind the score

A website can look finished in a browser and still reach ChatGPT as an empty page, and nothing tells the owner. I built CrawlReadable so you can see it in ten seconds, for free.

Every score comes from one plain fetch and rules anyone can read. If your report looks wrong, write to me and I'll check it myself.

Franco Smit, founder
Cape Town, South Africa
hello@crawlreadable.com

  • The methodology is public

    Every check, its weight and the grade bands are published. Read the methodology

  • The data is open

    Every score is free to download and reuse under CC BY 4.0. Download the CSV

  • Money-back guarantee

    Full refund if we miss the 2-day delivery, or if your score doesn't go up after you apply the fixes within 30 days. Refund policy

  • Payments by Paystack

    Card details go to Paystack and never reach us. No cookies, no ad trackers. Privacy policy

We'll write these fixes for you

Send us your domain and we'll write the fixes, ready to paste:

  • an llms.txt file written for your site
  • the robots.txt rules to let AI crawlers in
  • Organization and WebSite structured data (JSON-LD)
  • a plain-English note on anything only JavaScript shows, and how to fix it

Delivered by email within 2 business days. We re-score your site once you've applied it. See a sample fix pack.

$149

one site, one payment, charged as R2,450 (South African rand)

Get the fix pack

Or do it yourself: $49 (charged as R800). The same files, generated for your site and emailed within 1 business day, without the hand-written note or re-score. Get the DIY pack

Payments by Paystack

Money-back guarantee. Full refund if we miss the 2-day delivery, or if your score doesn't go up after you apply the fixes within 30 days. Refund policy

Questions

What does the CrawlReadable score measure?

It measures what an AI crawler that runs no JavaScript gets from your site: whether robots.txt lets GPTBot, ClaudeBot and PerplexityBot in, whether the HTML contains your actual text, whether the page describes itself with a title, description and structured data, and whether a sitemap lists your pages. 100 means nothing is in the way.

Do AI crawlers run JavaScript?

Mostly no. Analyses of crawler traffic, including one by Vercel and MERJ in December 2024, found that OpenAI's and Anthropic's crawlers download JavaScript files but don't execute them. Googlebot renders JavaScript, which is why a site can rank in Google and still be invisible to ChatGPT.

Is llms.txt worth adding?

It's cheap, so yes, but don't expect much from it alone. Server log studies in 2026 found that 97% of llms.txt files receive no requests from AI systems. Readable HTML and open robots.txt rules matter far more, which is why llms.txt is worth 4 of our 100 points.

Why did my site get no score?

Your server answered our fetcher with an error or a bot challenge page. That tells us nothing about how real AI crawlers are treated, so we don't guess. Firewall rules that only block certain bots by IP address are also invisible from outside.

What happens to my results?

Each scored audit gets a public page at crawlreadable.com/r/your-domain and a spot on the leaderboard. We store the score and what we fetched, nothing else. We don't set cookies.