Robots.txt Generator
Build a robots.txt file with per-crawler rules and a sitemap reference, ready to upload to your site's root.
Advertisement space
Frequently asked questions
- Yes, completely free with no signup or usage limit.
- Upload it as robots.txt at the root of your domain, e.g. https://example.com/robots.txt.
- Disallow tells crawlers not to access a path; Allow carves out an exception within a disallowed area, useful when you want to block a folder but allow one file inside it.
- It's a request that well-behaved crawlers respect, not a security measure — it doesn't prevent access, and some bots ignore it entirely.