Robots.txt Common Mistakes
robots.txt is easy to get subtly wrong. This lists the mistakes that most often cause crawl problems and how to avoid them.
Frequent mistakes
- Disallowing the whole site (
Disallow: /) by accident. - Blocking CSS/JS needed to render content.
- Assuming wildcard behavior is identical across all crawlers.
- No
Sitemap:directive.
Recommendations
- Keep rules minimal and intentional.
- Allow assets required for rendering.
- Include a Sitemap line.
- Test specific paths before shipping.
Verify
Test rules and a path with the Robots.txt Validator (simplified — not a full crawler). Background: improving crawlability.
FAQ
Is the validator a full crawler?
No — it is a simplified checker that approximates Allow/Disallow longest-match behavior.
Does Disallow remove a page from search?
Not necessarily; disallow blocks crawling, not indexing of already-known URLs. Use other controls for removal.