Why this matters
robots.txt is the standard mechanism for telling crawlers what they can and can't fetch. Without it, crawlers fall back to crawling everything they can find.
Background
A reachable /robots.txt tells crawlers which paths to skip + where the sitemap lives. Even an effectively-empty allow-all robots.txt is better than 404 — it signals intent + makes the file inspectable.
References
RFC 9309 (Robots Exclusion Protocol) · Google Search Central
How to fix
Code snippet for each stack we cover. Pick the one matching your server / framework.
nginx
Place a robots.txt file at the site root; nginx serves it as static.
apache
Same — DocumentRoot/robots.txt.
cloudflare
Pages / Workers can serve a Worker route returning the robots.txt content.
wordpress
Yoast / RankMath generate /robots.txt OR upload manually via FTP.
flask
Add @app.route('/robots.txt') returning a text response with User-agent: * + Allow / Disallow rules + Sitemap directive.
express
app.use('/robots.txt', express.static(...))
rails
Place in public/robots.txt — Rails serves it.
Verify it's working
curl -sI https://your-site/robots.txt — must return 200, not 404.
Want to know if your site has this issue?
Run a free audit — security, GDPR, technical SEO, AEO/GEO, and WCAG accessibility.
Audit my site →