Free AI Crawler Access Checker
Should you let AI crawlers in? Start with what your site is for
There is no universal answer, and anyone who gives you one is guessing. The right policy depends on what your site does and what you want from AI products.
A publisher or blog
Your words are your product. Many publishers block training crawlers such as GPTBot, ClaudeBot and CCBot, but allow AI search and user-triggered bots, so their pages can still be found and cited when someone asks an assistant a question.
A local business or shop
You usually want to be found. Blocking AI search bots may mean assistants cannot mention you. Training crawlers are a smaller worry when your pages mostly hold opening hours, prices and addresses.
A tool, SaaS or documentation site
Being quoted accurately can bring signups, so many sites in this group leave AI bots allowed and make sure the pages are clear and up to date.
Members-only or private content
Do not rely on robots.txt at all. It is a polite request, not a lock. Protect private material with logins.
Three kinds of bots, three different decisions
The word AI crawler hides a real difference. A training crawler collects pages that may be used to train future models. An AI search crawler builds an index that an AI search product uses to find and cite pages. A user-triggered fetcher only visits a page when a person asks an assistant to look at it, so blocking it means the assistant cannot read your page even when someone sends it the link.
That is why a blanket rule of block everything is rarely what people mean. If your aim is to stop your writing being used for training, you block the first kind. If your aim is to be recommended in AI answers, you need the second and third kinds to be able to reach you.
Reading the results table
Paste your robots.txt, leave the path as / to test the whole site, and read each row. Own group means your file names that bot. Falls back to * group means the bot is not named, so your general rules apply to it. The Used for column tells you whether a bot trains models, builds an AI search index, or only fetches a page when a user asks.
The classic trap: a bot with its own group ignores the * group entirely. Block GPTBot from one folder only, and it may crawl everything else, including folders your * group blocks.
Writing the rules once you have decided
If you choose to block only the training crawlers, a typical block looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /Bots that are not named, such as AI search and user-triggered bots, fall back to your star group and stay allowed if it allows them. Paste the finished file here to confirm that each row shows what you meant. The AI crawler robots.txt builder will write this for you if you prefer toggles to typing.
What robots.txt cannot do
It is a request, and reputable companies follow it. It does not remove content that was already collected, it cannot stop a bot that ignores the rules, and it cannot tell a real GPTBot from a script pretending to be one. If you need enforcement, your server logs show which user agents are actually visiting, and a firewall or a service in front of your site can block or verify bots by their network addresses. For most sites, robots.txt is a sensible first step and nothing more.
How to see which bots really visit you
Your robots.txt says what bots should do. Your server logs show what they actually do. Most hosting dashboards offer access logs or a statistics page where you can search for user agents such as GPTBot, ClaudeBot, PerplexityBot or Bytespider and see how often they came and which pages they asked for. If a bot you blocked is still crawling heavily, either the rule has a mistake, which the checker above will reveal, or the bot is ignoring the rule. In that case a firewall rule is the stronger tool.
A quick routine after every change
- Paste the new file here and read every row.
- Test your home page and one important inner page with the robots.txt tester, using Googlebot, to make sure search is unaffected.
- Look at the file live in your browser to confirm the right version was uploaded.
- Set a reminder to revisit it every few months, because new crawlers appear. The AI bot user-agent list helps you spot new names.
FAQ
Blocking AI training bots, including Google-Extended, does not affect Google Search. Blocking Googlebot would.
The major companies say their crawlers do. Smaller or unknown bots may not.
No. It is checked in your browser.
Twenty-four well-known crawlers, including those from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance and Common Crawl.