AI and search crawlers: the directory
The robots.txt names that matter for AI search and AI training, who runs them and what blocking each one does.
| Token | Operator | Type |
|---|---|---|
OAI-SearchBot | OpenAI | AI search |
GPTBot | OpenAI | AI training |
ChatGPT-User | OpenAI | User-requested fetch |
Claude-SearchBot | Anthropic | AI search |
ClaudeBot | Anthropic | AI training |
Claude-User | Anthropic | User-requested fetch |
PerplexityBot | Perplexity | AI search |
Perplexity-User | Perplexity | User-requested fetch |
Googlebot | Search engine | |
Google-Extended | AI training | |
Applebot | Apple | Search engine |
Applebot-Extended | Apple | AI training |
Bingbot | Microsoft | Search engine |
CCBot | Common Crawl | AI training / datasets |
Read the AI crawlers and robots.txt guide for a ready-made robots.txt that allows AI search and blocks AI training.
Last checked against the vendors' public documentation on November 17, 2026. Crawler names and policies change; follow the source links for the current rules.