Robots.txt Generator
Draft crawler rules and an optional sitemap directive. This tool runs in your browser and does not need an API key.
Use the Robots.txt Generator
Draft crawler rules and an optional sitemap directive.
Result
How to use this tool
Add allowed or blocked paths on separate lines. Optionally enter your sitemap URL and review the draft before publishing.
Frequently Asked Questions
Does robots.txt secure private pages?
No. It gives crawler instructions and is public. Protect private pages with access controls.
Where should I upload robots.txt?
For most sites, the file belongs in the website root so it opens at your domain followed by `/robots.txt`.
Does robots.txt stop a page from being indexed?
Not by itself. It gives crawlers instructions about crawling; a blocked URL may still be discovered or indexed from other links.
Can a robots.txt rule block my whole site?
Yes. A broad disallow rule can prevent crawlers from accessing pages, so review the generated rules before publishing.