AI crawler index

What is Bingbot?

Bingbot is Microsoft's search engine crawler. Bing's standard crawler, which does most of Bing's daily crawling.

At a glance

Should you block Bingbot?

It is a classic search engine crawler. Blocking it removes you from that search engine and from the products built on its index, so most sites should allow it.

Bing's index also feeds the web results of other products, so blocking Bingbot has a wider reach than Bing itself. Bing asks you not to hardcode its IPs; the list changes.

How to block or allow Bingbot in robots.txt

Block it everywhere:

User-agent: Bingbot
Disallow: /

Allow it everywhere:

User-agent: Bingbot
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

How to verify that a visit is really Bingbot

Anyone can write Bingbot in a user agent, so the user agent alone proves nothing. Take the IP address of the request from your server logs and check it.

Look up the hostname of the IP, then look up that hostname. The first answer must end in .search.msn.com, and the second must return the same IP; anything else is an impostor. On Windows, use nslookup instead of host.

host 203.0.113.7
host <hostname>

Microsoft publishes its IP ranges as JSON. This command downloads the list and prints True if the IP is in it. Replace 203.0.113.7 with the IP from your logs; Python 3 comes with macOS and most Linux systems.

curl -s https://www.bing.com/toolbox/bingbot.json \
  | python3 -c 'import sys, json, ipaddress as ip
a = ip.ip_address(sys.argv[1])
ranges = json.load(sys.stdin)["prefixes"]
print(any(a in ip.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges))' 203.0.113.7

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.