AI crawler index

What is Bytespider?

Bytespider is a crawler run by ByteDance. ByteDance does not publish documentation for Bytespider that we could verify.

At a glance

Operator	ByteDance
Purpose	Other or undeclared
User agent	Not documented by the operator
Obeys robots.txt	Not documented by the operator
Published IP ranges	Not documented by the operator
How to verify it	Not documented by the operator
Official documentation	Not documented by the operator

Should you block Bytespider?

Its purpose is not clearly documented. Check your server logs to see how often it visits and what it requests before deciding.

Without an official page, its purpose and whether it honors robots.txt are not documented. If you want it gone, add a robots.txt rule and confirm in your logs that the requests stop.

How to block or allow Bytespider in robots.txt

Block it everywhere:

User-agent: Bytespider
Disallow: /

Allow it everywhere:

User-agent: Bytespider
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

How to verify that a visit is really Bytespider

ByteDance publishes no IP list or hostname for Bytespider, so you cannot prove a visit is genuine. Treat the user agent as a claim and judge the visits by what they request in your logs.

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.

What our data says about Bytespider

Figures we measured on real AI answers and websites. Each one links to its study, with the method, the sample and the date.

17.3%

17.3% of the websites AI cites most in Spain block Bytespider from crawling their site, and 15.2% name it in a robots.txt group of its own.

Sample: 382 websites AI cites for three or more of 43 brands in Spain. Source: Mencoro (October 2026), in Which AI crawlers do the websites AI cites let in?