AI crawler index
What is ClaudeBot?
ClaudeBot is Anthropic's crawler for AI training. Collects web content that may contribute to training Anthropic's generative AI models.
At a glance
| Operator | Anthropic |
|---|---|
| Purpose | AI training |
| User agent | Not documented by the operator |
| Obeys robots.txt | Yes, according to the operator |
| Published IP ranges | https://claude.com/crawling/bots.json |
| How to verify it | Anthropic says a source IP on its published list indicates the crawler is from Anthropic. |
| Official documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
Should you block ClaudeBot?
It collects pages for model training, not for answering questions in real time. Blocking it keeps your future content out of that operator's training data, but it does not take you out of AI search answers, which other crawlers feed. Block it if you do not want your content in training sets; allow it if you want future models to know your brand.
Blocking it keeps your future content out of Anthropic's training data. It supports Crawl-delay, and Anthropic advises against IP blocking, because the bot then cannot read your robots.txt.
How to block or allow ClaudeBot in robots.txt
Block it everywhere:
User-agent: ClaudeBot
Disallow: /Allow it everywhere:
User-agent: ClaudeBot
Allow: /Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.
Other Anthropic crawlers
- Claude-SearchBot: AI search
- Claude-User: User-triggered fetch
Check your own site
The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.
FAQ
ClaudeBot questions
Start tracking your brand in AI search today
Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.