CCBot
The crawler for Common Crawl, a nonprofit that publishes a free, open archive of web pages. Many researchers and AI developers use that archive.
| Operated by | Common Crawl |
|---|---|
| Type | AI training / datasets |
| robots.txt token | CCBot |
| Official docs | Common Crawl's crawler documentation |
What blocking it does
Blocking CCBot keeps future pages out of new Common Crawl archives. It doesn't remove pages from archives that were already published.
robots.txt rules
To allow it everywhere:
User-agent: CCBot
Allow: /To block it everywhere:
User-agent: CCBot
Disallow: /
Put these in the robots.txt file at the root of your site. A group that names CCBot replaces your User-agent: * rules for that crawler, so copy any rules you still want into its group.
Questions
What is CCBot?
The crawler for Common Crawl, a nonprofit that publishes a free, open archive of web pages. Many researchers and AI developers use that archive.
What happens if I block CCBot?
Blocking CCBot keeps future pages out of new Common Crawl archives. It doesn't remove pages from archives that were already published.
Does allowing CCBot guarantee my business shows up?
No. Allowing a crawler only lets it read your pages. Whether Common Crawl's products show or recommend your business depends on many things nobody outside Common Crawl can see.
Not affiliated with, endorsed by or sponsored by OpenAI, Anthropic, Google, Apple, Microsoft, Perplexity or Common Crawl. ChatGPT is a trademark of OpenAI. Other names belong to their owners and are used only to identify their crawlers.
Last checked against the vendors' public documentation on November 17, 2026. Crawler names and policies change; follow the source links for the current rules.