Robots.txt generator
List the paths crawlers should skip, add your sitemap address and tick the AI crawlers you want to keep out, and this tool writes a robots.txt file you can copy or download. There’s also an option to block everything, for a staging site.
It’s for makers setting up a new site or deciding how they feel about AI crawlers. Each AI crawler is labelled with its company and whether it feeds search answers or model training, so you can block one without the other.
Free with a ShipScroll account.
How to use the Robots.txt Generator
- List paths to keep crawlers out of, one per line, such as /admin/ or /cart.
- Add your sitemap’s full address.
- Tick any AI crawlers you want to block from the whole site.
- Press Build robots.txt, then download the file and put it at the root of your site, as /robots.txt.
Search crawlers and training crawlers
Several AI companies run separate crawlers for different jobs. OpenAI’s GPTBot and Anthropic’s ClaudeBot collect pages for model training, while OAI-SearchBot and Claude-SearchBot fetch pages to answer searches and cite sources. Google-Extended controls whether Google uses your content for its AI models, and doesn’t affect Google Search. Blocking a training crawler while allowing the search one keeps you visible in AI answers.
What robots.txt can and can’t do
- It asks crawlers not to fetch paths. Reputable crawlers follow it, but it isn’t a lock.
- A blocked page can still appear in search if other sites link to it. Use noindex to keep a page out of results.
- Don’t block CSS or JavaScript your pages need, or search engines may not see the page as visitors do.
- Test your file afterwards with the Robots.txt Tester.
Questions
Where does the file go?
At the root of your domain, so it loads at https://yourdomain.com/robots.txt. Crawlers don’t look for it anywhere else, and each subdomain needs its own.
Will blocking AI crawlers remove me from ChatGPT or Claude?
Blocking a training crawler stops future collection for training. Answers that search the web use the search crawlers, which you can leave allowed. Content already collected isn’t removed by a robots.txt change.
Should I block /admin?
It does no harm, but anything private should need a login. Listing a path in robots.txt also tells anyone reading the file that it exists.
Is the file I build saved anywhere?
No. It’s built in your browser. Copy or download it before you leave the page.
Related tools
- Robots.txt TesterCheck whether a crawler may fetch a page, and which robots.txt rule decides it.
- AI Visibility CheckerSee which AI crawlers your robots.txt allows, plus llms.txt and schema readiness.
- Sitemap CheckerFind a site’s sitemap, count its URLs, and catch errors against the protocol.