AI crawler index
What is Bytespider?
Bytespider is a crawler run by ByteDance. ByteDance does not publish documentation for Bytespider that we could verify.
At a glance
| Operator | ByteDance |
|---|---|
| Purpose | Other or undeclared |
| User agent | Not documented by the operator |
| Obeys robots.txt | Not documented by the operator |
| Published IP ranges | Not documented by the operator |
| How to verify it | Not documented by the operator |
| Official documentation | Not documented by the operator |
Should you block Bytespider?
Its purpose is not clearly documented. Check your server logs to see how often it visits and what it requests before deciding.
Without an official page, its purpose and whether it honors robots.txt are not documented. If you want it gone, add a robots.txt rule and confirm in your logs that the requests stop.
How to block or allow Bytespider in robots.txt
Block it everywhere:
User-agent: Bytespider
Disallow: /Allow it everywhere:
User-agent: Bytespider
Allow: /Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.
Check your own site
The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.
FAQ
Bytespider questions
Start tracking your brand in AI search today
Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.