AI crawler index
What is Googlebot?
Googlebot is Google's search engine crawler. Builds Google's search indexes for Search (including Discover), Images, Video and News.
At a glance
| Operator | |
|---|---|
| Purpose | Search engine |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 |
| Obeys robots.txt | Yes, according to the operator |
| Published IP ranges | https://developers.google.com/static/crawling/ipranges/common-crawlers.json |
| How to verify it | Reverse DNS to crawl-*.googlebot.com or geo-crawl-*.geo.googlebot.com and forward-confirm, or match Google's IP list. |
| Official documentation | https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers |
Should you block Googlebot?
It is a classic search engine crawler. Blocking it removes you from that search engine and from the products built on its index, so most sites should allow it.
Blocking Googlebot removes you from Google Search, Discover, Images, Video and News. W.X.Y.Z stands for the Chrome version, so match it with a wildcard. There is also a smartphone variant.
How to block or allow Googlebot in robots.txt
Block it everywhere:
User-agent: Googlebot
Disallow: /Allow it everywhere:
User-agent: Googlebot
Allow: /Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.
Other Google crawlers
- Google-Extended: robots.txt control token
- GoogleOther: Other or undeclared
Check your own site
The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.
FAQ
Googlebot questions
Start tracking your brand in AI search today
Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.