SEO
Robots.txt Generator
Free robots.txt generator — build a valid robots.txt with allow/disallow rules, user-agent directives, and your sitemap URL.
robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /private Sitemap: https://example.com/sitemap.xml
How to use Robots.txt Generator
- Choose which bots to allow or disallow.
- Add your sitemap URL.
- Copy the generated robots.txt into your site root.
Frequently asked questions
Where do I put robots.txt?
At the root of your domain: https://example.com/robots.txt
Should I block AI crawlers like GPTBot?
Only if you don't want your content used for training. Blocking hurts AI-search visibility (AEO/GEO) — many sites now allow AI bots deliberately.