geo
How to fix: AI crawlers (GPTBot / ClaudeBot / etc.) not blocked
Why this matters
If GPTBot / ClaudeBot / PerplexityBot / CCBot are blocked in robots.txt, the site cannot be indexed as a training OR retrieval source for any LLM product using those crawlers. Zero LLM citation visibility regardless of on-page quality.
Background
If GPTBot / ClaudeBot / PerplexityBot / CCBot are blocked in robots.txt, the site cannot be indexed as a training OR retrieval source for any LLM product using those crawlers. Zero LLM citation visibility regardless of on-page quality.
References
OpenAI GPTBot · Anthropic ClaudeBot · CCBot · Perplexity · Common Crawl
How to fix
Code snippet for each stack we cover. Pick the one matching your server / framework.
nginx
No server config — robots.txt edit.
apache
Same.
cloudflare
Cloudflare 'AI Audit' / 'Bot Management' may also block AI crawlers at the edge — check Settings → Bots if your robots.txt looks clean but bots are still blocked.
wordpress
Edit robots.txt (often via Yoast / RankMath). Remove specific blocks for GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, CCBot, GoogleOther, Google-Extended unless you have a deliberate reason to block.
flask
Update your robots.txt template — no User-agent: GPTBot followed by Disallow: /.
express
Same.
rails
Same.
Verify it's working
curl https://your-site/robots.txt | grep -iE 'gptbot|claude|perplexity|ccbot' — should NOT see Disallow: /.
Want to know if your site has this issue?
Run a free 53-check audit — security, GDPR, NIS2, and technical SEO.
Audit my site →