AI crawler index

What is CCBot?

CCBot is Common Crawl's crawler for AI training. Builds Common Crawl's free, open repository of web crawl data, a common source for training datasets.

At a glance

Should you block CCBot?

It collects pages for model training, not for answering questions in real time. Blocking it keeps your future content out of that operator's training data, but it does not take you out of AI search answers, which other crawlers feed. Block it if you do not want your content in training sets; allow it if you want future models to know your brand.

It obeys Crawl-delay. Blocking it stops future crawls only, not the archives already published. Common Crawl warns that fake CCBots exist, so verify it.

How to block or allow CCBot in robots.txt

Block it everywhere:

User-agent: CCBot
Disallow: /

Allow it everywhere:

User-agent: CCBot
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

Test CCBot on your robots.txt

How to let CCBot through a firewall or CDN

robots.txt only tells CCBot what it may fetch. If a firewall, a CDN or a bot-protection rule blocks the request first, it never reads the file, so allowing it in robots.txt is not enough.

Allow the IP ranges Common Crawl publishes (https://index.commoncrawl.org/ccbot.json) rather than the user agent, which anyone can fake.

If your firewall can match hostnames, allow requests whose reverse DNS ends in .crawl.commoncrawl.org and resolves back to the same IP.

How to verify that a visit is really CCBot

Anyone can write CCBot in a user agent, so the user agent alone proves nothing. Take the IP address of the request from your server logs and check it.

Look up the hostname of the IP, then look up that hostname. The first answer must end in .crawl.commoncrawl.org, and the second must return the same IP; anything else is an impostor. On Windows, use nslookup instead of host.

host 203.0.113.7
host <hostname>

Common Crawl publishes its IP ranges as JSON. This command downloads the list and prints True if the IP is in it. Replace 203.0.113.7 with the IP from your logs; Python 3 comes with macOS and most Linux systems.

curl -s https://index.commoncrawl.org/ccbot.json \
  | python3 -c 'import sys, json, ipaddress as ip
a = ip.ip_address(sys.argv[1])
ranges = json.load(sys.stdin)["prefixes"]
print(any(a in ip.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges))' 203.0.113.7

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.