Home / AI Search Tools / AI Crawler Robots.txt Builder
AI Crawler Robots.txt Builder
Choose which AI crawlers may visit your site and get a ready-to-use robots.txt, with one-click presets for training bots.
Presets:
| Bot | Purpose | Rule |
|---|
robots.txt is a request, not a lock. Well-behaved bots follow it, but it cannot force a crawler to obey. If you already have a robots.txt, merge these groups into it instead of replacing it. Bot names and purposes change, so check each company documentation for the latest.
Training, search and user bots
AI companies run different bots for different jobs. Training bots collect content that may be used to build models. Search bots index pages so they can appear in AI search results. User bots fetch a page only when a person asks an assistant about it. You can allow some and block others.
How to use this builder
Pick a preset or set each bot to Allow or Block. The robots.txt on the right updates as you go. Copy or download it and add the rules to the robots.txt at the root of your site.
Blocking has trade-offs
Blocking search bots can reduce how often your site appears in AI answers, while blocking training bots mostly affects whether your content is used to build future models. Robots.txt relies on bots following the rules, so it will not stop a crawler that ignores them.
FAQ
Blocking AI bots does not block Googlebot, so normal Google search crawling continues. Google-Extended is a separate control that covers use of your content for Gemini, and Google says it does not affect inclusion in Google Search.
Reputable ones say they do, but robots.txt is a voluntary standard. For stronger protection use server or firewall rules.