Home / AI Search Tools / AI Bot User-Agent List
AI Bot User-Agent List
A searchable list of the main AI crawler user-agents, who runs them and what they are used for.
| User-agent | Operator | Purpose | What it does |
|---|
Training bots collect content that may be used to build AI models. Search bots index pages so they can appear in AI search results. User request bots fetch a page only when a person asks an AI assistant about it. Lists change often, so confirm against each company documentation before you rely on them.
How to read this list
Each row shows the user-agent name you would put in robots.txt, the company behind it and its main purpose. Training bots gather content for model training, search bots build AI search indexes and user request bots fetch pages on demand.
Keep the list current
AI companies add and rename crawlers regularly. Check the official documentation from each company before you block or allow a bot, and review your robots.txt every few months.
Next step
When you know which bots you want to allow or block, use the AI Crawler Robots.txt Builder to create the rules.
FAQ
Check your server access logs or your hosting analytics for these user-agent names.
No. It covers the main AI crawlers, and new ones appear often.