A few years ago, robots.txt was mostly about Google and Bing. Now a growing list of AI companies run their own crawlers, and site owners have a real decision to make: who may read my pages, and for what purpose? There is no universal right answer. This guide explains what the different bots do, how to choose for your kind of site, and how to write and test the rules.
Three kinds of AI bots
The phrase AI crawler hides an important difference.
- Training bots collect public pages that may be used to train future models. Examples include GPTBot, ClaudeBot and CCBot.
- Search bots build an index that an AI search product uses to find and cite pages.
- User request bots fetch a page only when a person asks an assistant to open or read it. Blocking them means the assistant cannot read your page even when someone sends it the link.
Blocking everything is rarely what people mean. If your aim is to stop your writing being used for training, you block the first kind. If you want to be found and cited in AI answers, the second and third kinds need to reach you. The AI bot user-agent list labels each bot with its purpose.
Control tokens are not crawlers
Google-Extended and Applebot-Extended are tokens you can put in robots.txt to say whether content collected by those companies’ normal crawlers may be used for AI training. They do not appear as visitors in your logs. Blocking Google-Extended does not stop Googlebot, and your pages stay in Google Search.
Choosing for your kind of site
A publisher or blog
Your words are your product. Many publishers block training crawlers but allow search and user bots, so their pages can still be found and cited.
A local business or shop
You usually want to be found. Blocking search bots may mean assistants cannot mention you. Training crawlers are a smaller worry when your pages mostly hold hours, prices and addresses.
A tool, software or documentation site
Being quoted accurately can bring signups, so many sites in this group leave AI bots allowed and keep their pages clear and current.
Private or members-only content
Do not rely on robots.txt at all. It is a polite request, not a lock. Protect private material with logins.
A sample file
If you choose to block only training crawlers, a typical block looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlBots that are not named fall back to the star group and stay allowed if it allows them. The AI crawler robots.txt builder writes this with toggles, and the robots.txt generator builds a general file. Merge new groups into your existing file instead of replacing it.
The trap that catches most people
A bot with its own group ignores the star group completely. If you add a group for GPTBot that blocks one folder, GPTBot follows only that group, and everything else is allowed, including folders that your star group blocks. Repeat any rules you still want inside the named group.
Test before you trust
Paste your finished file into the AI crawler access checker to see, bot by bot, what is allowed, and use the robots.txt tester to confirm that Googlebot can still reach your key pages. Then open yoursite.com/robots.txt in a browser to make sure the right version is live.
What robots.txt cannot do
- It is a request, followed by reputable companies, not a lock.
- It does not remove content that was already collected.
- It cannot stop a bot that pretends to be another bot. Your server logs show which user agents really visit, and a firewall can block bad ones.
- It does not remove a page from search results. For that, allow crawling and use a noindex tag.
llms.txt: a different file with a different job
llms.txt is a proposed file that summarises your site for language models in plain text. It is a community proposal, not an official standard, and it has no known effect on search rankings. The llms.txt generator and validator are there if you want one, but robots.txt, not llms.txt, controls who may crawl.
The short version
Decide what you want from AI products first, then write rules that match: block training bots if you do not want your content used that way, allow search and user bots if you want to be cited, test the result, and review the file every few months because new crawlers appear often.
Frequently asked questions
Does blocking AI bots hurt my Google rankings?
Blocking AI training tokens such as Google-Extended does not affect Google Search. Blocking Googlebot would, so check your rules before you publish them.
Do AI crawlers obey robots.txt?
The major companies say their crawlers do. Smaller or unknown bots may not, which is why server logs and a firewall matter for enforcement.
Should I block all AI bots?
Only if that matches your goals. Blocking search and user bots can reduce your visibility in AI answers, while many sites block only training bots.
Where does robots.txt go?
At the root of your domain, for example https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own.