JavaScript and AI crawlers: why single-page apps look empty

Most AI crawlers download your HTML and read it as is, without running JavaScript. If your site builds its content in the browser, as single-page apps built with React, Vue or Angular do by default, those crawlers receive an empty shell: a title, a script tag and an empty div.

Which crawlers run JavaScript

Googlebot renders JavaScript, in a second pass that can lag by days. The AI crawlers mostly don't: an analysis of crawler traffic by Vercel and MERJ in December 2024 found that OpenAI's and Anthropic's crawlers fetched JavaScript files but never executed them.

That's why a JavaScript site can rank fine in Google and still be invisible to ChatGPT, Claude and Perplexity.

How to check your site

Fetch the raw HTML and read it: curl -s https://yoursite.com | less. If you can't find your headline and product description in there, AI crawlers can't either.

CrawlReadable does this for you and shows the first 400 characters a no-JavaScript crawler gets. Under 400 characters of text counts as a fail.

How to fix it

Server-side rendering: Next.js, Nuxt, SvelteKit, Remix and Astro render pages on the server, so the HTML arrives complete. For an existing React app, moving to Next.js is the usual path.

Static generation: if pages don't change per visitor, build them to HTML at deploy time. Most of the frameworks above support it, and it's the fastest option.

Prerendering: a service renders your pages in a headless browser and serves the result to bots. It's a patch, not a fix, but it works without touching the app.

At minimum, put the essentials in the initial HTML: the title, meta description, h1 and a paragraph saying what the product does.

Questions