Ready for Agents

AI Crawlers and robots.txt: Which Bots to Allow or Block

The AI crawlers from OpenAI, Anthropic, Perplexity, Google and Apple, what each one does, and robots.txt rules to allow AI search while blocking training.

AI companies now run separate crawlers for different jobs. The big split: search crawlers fetch pages so an AI product can show and link to them, training crawlers collect pages that may be used to train models, and user-requested fetchers visit a page because someone asked the assistant to. You can treat each one differently in robots.txt.

TokenOperatorType
OAI-SearchBotOpenAIAI search
GPTBotOpenAIAI training
ChatGPT-UserOpenAIUser-requested fetch
Claude-SearchBotAnthropicAI search
ClaudeBotAnthropicAI training
Claude-UserAnthropicUser-requested fetch
PerplexityBotPerplexityAI search
Perplexity-UserPerplexityUser-requested fetch
GooglebotGoogleSearch engine
Google-ExtendedGoogleAI training
ApplebotAppleSearch engine
Applebot-ExtendedAppleAI training
BingbotMicrosoftSearch engine
CCBotCommon CrawlAI training / datasets

A starting robots.txt: allow AI search, opt out of training

# Allow AI search and regular search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /

# Opt out of AI training (optional; your choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Several user-agent lines in a row share the rules beneath them. If you're happy for your content to be used in training, leave the second block out. Replace the sitemap address with yours.

Things that trip people up

Last checked against the vendors' public documentation on November 17, 2026. Crawler names and policies change; follow the source links for the current rules.