Can AI crawlers read vanikya.ai?
AI crawlers miss most of this site. 9 of 20 checks failed.
Scored 43/100 on 2026-10-15 from https://vanikya.ai/
What an AI crawler gets
The first 400 characters of text in the HTML, before any JavaScript runs.
AI Creative Suite with a Self-Reviewing Agent | Vanikya
55 characters of readable text, under the 400 we count as readable
Which AI crawlers are allowed
| Crawler | robots.txt says |
|---|---|
| GPTBot | Blocked |
| OAI-SearchBot | Allowed |
| ClaudeBot | Blocked |
| Claude-SearchBot | Allowed |
| PerplexityBot | Allowed |
| Perplexity-User | Allowed |
| Google-Extended | Allowed |
| CCBot | Blocked |
| Bytespider | Blocked |
| Amazonbot | Allowed |
Score by category
Fixes, biggest first
Put your words in the HTML
A crawler that runs no JavaScript found under 400 characters of text. Whatever your page says after scripts run, it doesn't say to them.
Stop serving an empty app shell
The page is a JavaScript shell: scripts plus almost no text. To most AI crawlers it looks blank.
Let GPTBot crawl the site
Your robots.txt blocks OpenAI's crawler, so ChatGPT learns about you only from other sites.
Let ClaudeBot crawl the site
Your robots.txt blocks Anthropic's crawler, so Claude learns about you only from other sites.
Add real copy to the page
Readable text is a tiny share of the page's code, or there's very little of it. Crawlers summarise words, not markup.
Give the page an h1 heading
One clear main heading tells a crawler what the page is about.
Use semantic HTML tags
Tags like main, nav, article and footer show which part is content and which is chrome.
Describe your organisation in JSON-LD
Organization or WebSite objects tie your site to a known entity, so answers name you correctly.
Mirror llms.txt at /.well-known/llms.txt
Some tools look in the .well-known folder instead. A copy of the same file covers both.
Get told when this changes
We're building monitoring: a re-score every week and an email when the score moves. Leave your email to get it first.
Show your score
Add this badge to your site or README. It always shows the latest score.
11 checks passed
- Publish a robots.txt file (4 points)
- Let PerplexityBot crawl the site (6 points)
- Add an llms.txt file (3 points)
- Add structured data (JSON-LD) (6 points)
- Write a descriptive page title (3 points)
- Add a meta description (3 points)
- Add Open Graph tags (2 points)
- Add a canonical link (2 points)
- Publish a sitemap.xml (6 points)
- Keep sitemap dates fresh (4 points)
- Offer a machine-readable interface (4 points)