AI Crawlers and robots.txt: Which Bots to Allow or Block
The AI crawlers from OpenAI, Anthropic, Perplexity, Google and Apple, what each one does, and robots.txt rules to allow AI search while blocking training.
AI companies now run separate crawlers for different jobs. The big split: search crawlers fetch pages so an AI product can show and link to them, training crawlers collect pages that may be used to train models, and user-requested fetchers visit a page because someone asked the assistant to. You can treat each one differently in robots.txt.
| Token | Operator | Type |
|---|---|---|
OAI-SearchBot | OpenAI | AI search |
GPTBot | OpenAI | AI training |
ChatGPT-User | OpenAI | User-requested fetch |
Claude-SearchBot | Anthropic | AI search |
ClaudeBot | Anthropic | AI training |
Claude-User | Anthropic | User-requested fetch |
PerplexityBot | Perplexity | AI search |
Perplexity-User | Perplexity | User-requested fetch |
Googlebot | Search engine | |
Google-Extended | AI training | |
Applebot | Apple | Search engine |
Applebot-Extended | Apple | AI training |
Bingbot | Microsoft | Search engine |
CCBot | Common Crawl | AI training / datasets |
A starting robots.txt: allow AI search, opt out of training
# Allow AI search and regular search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
Allow: /
# Opt out of AI training (optional; your choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Several user-agent lines in a row share the rules beneath them. If you're happy for your content to be used in training, leave the second block out. Replace the sitemap address with yours.
Things that trip people up
- A named group replaces the
*group for that crawler. If you block/admin/for everyone, repeat it in the named groups. - Some hosts and CDNs block AI bots with firewall rules regardless of robots.txt. Check your host's bot settings too.
- User-requested fetchers like ChatGPT-User and Perplexity-User may not follow robots.txt, according to their operators.
- Changes aren't instant. OpenAI says about 24 hours for its search crawler; Perplexity says up to 24 hours.
Last checked against the vendors' public documentation on November 17, 2026. Crawler names and policies change; follow the source links for the current rules.