AI crawler index

What is Baiduspider?

Baiduspider is Baidu's search engine crawler. Baidu's crawler, which visits websites automatically to collect their content for Baidu Search.

At a glance

Operator	Baidu
Purpose	Search engine
User agent	`Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)`
Obeys robots.txt	Yes, according to the operator
Published IP ranges	Not documented by the operator
How to verify it	Reverse DNS must end in baidu.com or baidu.jp, then a forward lookup must return the same IP.
Official documentation	https://ziyuan.baidu.com/college/articleinfo?id=1002

Should you block Baiduspider?

It is a classic search engine crawler. Blocking it removes you from that search engine and from the products built on its index, so most sites should allow it.

Baidu says it will not crawl, index or show pages blocked by robots.txt. The mobile user agent is longer but ends in the same Baiduspider/2.0 part, and Baidu says any hostname outside baidu.com or baidu.jp is an impostor.

How to block or allow Baiduspider in robots.txt

Block it everywhere:

User-agent: Baiduspider
Disallow: /

Allow it everywhere:

User-agent: Baiduspider
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

Test Baiduspider on your robots.txt

How to let Baiduspider through a firewall or CDN

robots.txt only tells Baiduspider what it may fetch. If a firewall, a CDN or a bot-protection rule blocks the request first, it never reads the file, so allowing it in robots.txt is not enough.

If your firewall can match hostnames, allow requests whose reverse DNS ends in .baidu.com or .baidu.jp and resolves back to the same IP.

How to verify that a visit is really Baiduspider

Anyone can write Baiduspider in a user agent, so the user agent alone proves nothing. Take the IP address of the request from your server logs and check it.

Look up the hostname of the IP, then look up that hostname. The first answer must end in .baidu.com or .baidu.jp, and the second must return the same IP; anything else is an impostor. On Windows, use nslookup instead of host.

host 203.0.113.7
host <hostname>

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.

What our data says about Baiduspider

Figures we measured on real AI answers and websites. Each one links to its study, with the method, the sample and the date.

7.1%

7.1% of the websites AI cites most in Spain block Baiduspider from crawling their site, and 8.3% name it in a robots.txt group of its own.

Sample: 678 websites AI cites for three or more of 43 brands in Spain. Source: Mencoro (November 2026), in Which AI crawlers do the websites AI cites let in?