tech

How to fix: robots.txt syntactically valid

Why this matters

A malformed robots.txt is ignored entirely by some crawlers — meaning your intended rules don't apply. The file is plain text but the syntax is strict.

Background

robots.txt syntax errors (typos, missing colons, wrong User-agent format) cause whole rules to be silently ignored by crawlers. A 'Disallow: /' meant for staging that became syntactically invalid in prod = unintended-everything-allowed.

References

RFC 9309 §2 · Google Search Central robots.txt syntax

How to fix

Code snippet for each stack we cover. Pick the one matching your server / framework.

nginx

No server config — fix the robots.txt content.

apache

Same.

cloudflare

Same.

wordpress

Validate via Google Search Console → robots.txt Tester. Fix flagged lines.

flask

Validate via https://www.google.com/webmasters/tools/robots-testing-tool.

express

Same.

rails

Same.

Verify it's working

Use Google Search Console robots.txt Tester — should report 0 errors + 0 warnings.

Want to know if your site has this issue?

Run a free 53-check audit — security, GDPR, NIS2, and technical SEO.

Audit my site →