tech

How to fix: Indexing not blocked by robots / X-Robots-Tag

Why this matters

A <meta name="robots" content="noindex"> tag tells crawlers not to include the page in their index. On a homepage that's almost always a mistake — usually left over from a staging deploy.

Background

A <meta name="robots" content="noindex"> tag or X-Robots-Tag: noindex header tells search engines to drop the page from their index. On a homepage that's almost always a mistake — usually left over from staging.

References

Google Search Central · meta robots + X-Robots-Tag

How to fix

Code snippet for each stack we cover. Pick the one matching your server / framework.

nginx

Remove `add_header X-Robots-Tag noindex;` if present.

apache

Same — remove the Header set X-Robots-Tag noindex directive.

cloudflare

Transform Rules — remove any X-Robots-Tag: noindex set on production hostnames.

wordpress

Settings → Reading → uncheck 'Discourage search engines'. Check Yoast / RankMath per-page noindex toggles.

flask

Remove <meta name="robots" content="noindex"> from your base template. Audit X-Robots-Tag headers set in middleware.

express

Same.

rails

Same.

Verify it's working

curl -sI https://your-site/ | grep -i x-robots-tag — should be missing or non-noindex. View source — no <meta name=robots content=noindex>.

Want to know if your site has this issue?

Run a free 53-check audit — security, GDPR, NIS2, and technical SEO.

Audit my site →