Generator
robots.txt Generator for SEO Crawl Rules
Generate an SEO-friendly robots.txt with crawl rules, sitemap links, and presets for blogs, docs, and SaaS sites.
What is robots.txt?
The robots.txt file lives at the root of your website and tells
search engine crawlers which pages they can and cannot access. It is not a security
mechanism — it is a polite request that well-behaved bots follow.
Should I block AI crawlers?
Many site owners now choose to block AI training crawlers like GPTBot, Google-Extended, and CCBot to prevent their content from being used to train language models. Our "Block AI crawlers" preset adds these rules automatically.