Can AI crawlers read wikipedia.org?
AI crawlers can read some of this site. The fixes below close the gaps. 10 of 20 checks failed.
Scored 63/100 on 2026-10-14 from https://www.wikipedia.org/
What an AI crawler gets
The first 400 characters of text in the HTML, before any JavaScript runs.
Wikipedia Wikipedia The Free Encyclopedia English 7,249,000+ articles 日本語 1,521,000+ 記事 Deutsch 3.156.000+ Artikel Français 2 783 000+ articles Русский 2 121 000+ статей Español 2.141.000+ artículos Italiano 1.989.000+ voci 中文 1,559,000+ 条目 / 條目 Português 1.184.000+ artigos Polski 1 710 000+ haseł Search Wikipedia Afrikaans Shqip العربية Asturianu Azərbaycanca Български 閩南語 / Bân-lâm-gú বাংলা Бела...
8,755 characters of readable text
Which AI crawlers are allowed
| Crawler | robots.txt says |
|---|---|
| GPTBot | Allowed |
| OAI-SearchBot | Allowed |
| ClaudeBot | Allowed |
| Claude-SearchBot | Allowed |
| PerplexityBot | Allowed |
| Perplexity-User | Allowed |
| Google-Extended | Allowed |
| CCBot | Allowed |
| Bytespider | Allowed |
| Amazonbot | Allowed |
Score by category
Fixes, biggest first
Add structured data (JSON-LD)
Machine-readable facts about your business: name, logo, what you sell. AI systems use it to describe you accurately.
Publish a sitemap.xml
A list of every page you want found. Without one, crawlers only find what's linked.
Add real copy to the page
Readable text is a tiny share of the page's code, or there's very little of it. Crawlers summarise words, not markup.
Describe your organisation in JSON-LD
Organization or WebSite objects tie your site to a known entity, so answers name you correctly.
Keep sitemap dates fresh
No lastmod date in the last 90 days, so crawlers can't tell what changed and may not come back.
Offer a machine-readable interface
A valid /openapi.json or /llms-full.txt lets AI agents use your product or read your docs in full, not just your homepage.
Add an llms.txt file
A short Markdown summary of your site at /llms.txt. Cheap to add, but few AI systems request it yet, so it's worth 3 points.
Write a descriptive page title
The title is often the only line an AI answer quotes. Say what you are and who it's for, in 10 to 100 characters.
Add a canonical link
Tells crawlers which URL is the real one when the same page is reachable at several.
Mirror llms.txt at /.well-known/llms.txt
Some tools look in the .well-known folder instead. A copy of the same file covers both.
Get told when this changes
We're building monitoring: a re-score every week and an email when the score moves. Leave your email to get it first.
Show your score
Add this badge to your site or README. It always shows the latest score.
10 checks passed
- Publish a robots.txt file (4 points)
- Let GPTBot crawl the site (6 points)
- Let ClaudeBot crawl the site (6 points)
- Let PerplexityBot crawl the site (6 points)
- Put your words in the HTML (18 points)
- Stop serving an empty app shell (10 points)
- Give the page an h1 heading (4 points)
- Use semantic HTML tags (4 points)
- Add a meta description (3 points)
- Add Open Graph tags (2 points)