Can ChatGPT crawl React-based websites?

Yes, but with significant limitations. While it can execute some JavaScript, it often times out or fails to wait for async API calls to complete. If your content isn't in the initial HTML, there is a high risk it won't be indexed for citations.

How do I know if an AI bot can see my JS content?

The easiest test is to disable JavaScript in your browser and reload the page. If the content disappears, AI bots likely can't see it either. You can also use our AI Search Readiness Tool to simulate a bot crawl.

Does Next.js solve this automatically?

Only if you use Server Components or SSR (getServerSideProps). If you use "use client" for your main content blocks and fetch data inside a useEffect, you are still vulnerable to the "blank page" problem for AI bots.

Why AI Crawlers Hate Your JavaScript (And How to Fix It)

One of the checks in my AI Search Readiness scanner compares what a page looks like with JavaScript enabled vs. without it. It loads the page both ways and compares the word count. The number of sites where AI crawlers see a completely blank page is surprisingly high.

I'm not talking about minor differences. I mean static HTML with literally zero words — just an empty <div id="root"></div> and a JavaScript bundle. The site looks great in a browser, but AI crawlers like GPTBot, OAI-SearchBot, and PerplexityBot don't have Google's rendering budget. They grab the HTML response and move on. If your content isn't there, it doesn't exist for them.

What Actually Happens During a CSR Crawl

I see this a lot with React SPAs and Vue apps. In a typical client-side rendering setup, your server sends a nearly empty HTML file and a bundle of JavaScript. To see the actual content, a crawler would need to:

Download the JS bundle
Execute the code
Fetch data from your API
Render the UI

Google invested years and massive infrastructure into doing exactly this. AI crawlers haven't. They have thousands of sites to process, limited compute per page, and no particular reason to wait for your React app to hydrate.

When I scan sites, my tool penalizes the Machine Readability score by 50% if the static HTML word count is 50 words or less. That penalty exists because I kept seeing sites score well on everything else — good schema, good content — but the content was invisible without JavaScript. A perfect score on content that no AI bot can read is meaningless.

Here's What I'd Check

1. Move to Server-Side Rendering (SSR) or Static Site Generation (SSG)

The most effective fix is making sure your core content is in the HTML before any JavaScript runs. If you're on Next.js, use Server Components or getStaticProps. If you're on Nuxt, enable SSR mode. The framework doesn't matter — the principle is the same: bake content into the response.

The easiest test I know: Turn off JavaScript in your browser (DevTools → Settings → Disable JavaScript) and reload your product page. If you see your product details, AI bots will too. If you see a spinner or a blank page, that's your problem right there.

2. Hydrate Metadata and JSON-LD First

Even if you can't move your entire UI to SSR, you must ensure your Schema.org (JSON-LD) data is present in the initial HTML. AI crawlers prioritize structured data over raw text. I've seen sites where the visible content was server-rendered but the JSON-LD was injected by a client-side script after an API call. That structured data is invisible to AI agents.

<!-- Correct: JSON-LD in <head>, server-rendered -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Your Product Name",
  "offers": {
    "@type": "Offer",
    "price": "29.99",
    "priceCurrency": "EUR"
  }
}
</script>

3. Check for Bot Blockers in Your Middleware

This one is easy to miss. Many WAFs and bot protection services (Cloudflare, Vercel's default settings) can block AI agents without you realizing it. I've scanned sites where robots.txt allowed everything, but Cloudflare's JS challenge stopped the bots before they ever reached the page. Check your robots.txt allows these bots, and verify your server doesn't challenge them with CAPTCHAs.

# robots.txt — allow all major AI crawlers
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

How to Test Your Site

A score below 60 in the Machine Readability dimension is a strong indicator that JS rendering is blocking visibility. Here's what I'd check first:

Disable JavaScript in browser DevTools → reload your key pages
Check your robots.txt for AI bot disallow rules
Verify JSON-LD is in the page source (Ctrl+U), not injected after load
Run a full AI Search Readiness audit to see the Machine Readability score

An honest caveat: Fixing JS rendering removes a real barrier — if AI crawlers can't read your page, nothing else matters. But I want to be upfront: I ran an empirical study on 441 domains and found essentially zero correlation (r=0.009) between structural readiness scores and actual LLM citations. Making your site machine-readable is a necessary prerequisite, not a citation guarantee. Think of it as removing a blocker, not flipping a switch.

You can use my free AI Search Readiness audit to simulate an AI crawl and see exactly which signals your site is missing — including the JS rendering check. For the full list of optimizations, see the AI Search Readiness checklist.

Why AI Crawlers Hate Your JavaScript (And How to Fix It)

What Actually Happens During a CSR Crawl

Here's What I'd Check

1. Move to Server-Side Rendering (SSR) or Static Site Generation (SSG)

2. Hydrate Metadata and JSON-LD First

3. Check for Bot Blockers in Your Middleware

How to Test Your Site

Frequently Asked Questions

Check Your AI Search Readiness

Related Articles

Free AI Search Readiness Audit Tool — Check Your Site in 2 Minutes

Schema.org Markup for AI Search Visibility: E-Commerce Guide

What Blocks Your Products from Appearing in Perplexity Answers

How to Make Your Documentation AI-Agent Ready in 2026