Ready for Agents

AI and search crawlers: the directory

The robots.txt names that matter for AI search and AI training, who runs them and what blocking each one does.

TokenOperatorType
OAI-SearchBotOpenAIAI search
GPTBotOpenAIAI training
ChatGPT-UserOpenAIUser-requested fetch
Claude-SearchBotAnthropicAI search
ClaudeBotAnthropicAI training
Claude-UserAnthropicUser-requested fetch
PerplexityBotPerplexityAI search
Perplexity-UserPerplexityUser-requested fetch
GooglebotGoogleSearch engine
Google-ExtendedGoogleAI training
ApplebotAppleSearch engine
Applebot-ExtendedAppleAI training
BingbotMicrosoftSearch engine
CCBotCommon CrawlAI training / datasets

Read the AI crawlers and robots.txt guide for a ready-made robots.txt that allows AI search and blocks AI training.

Last checked against the vendors' public documentation on November 17, 2026. Crawler names and policies change; follow the source links for the current rules.