AI crawler index

What is GoogleOther?

GoogleOther is a crawler run by Google. A generic crawler Google product teams use for one-off fetches, such as internal research and development.

At a glance

Operator	Google
Purpose	Other or undeclared
User agent	`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36`
Obeys robots.txt	Yes, according to the operator
Published IP ranges	https://developers.google.com/static/crawling/ipranges/common-crawlers.json
How to verify it	Reverse DNS to crawl-.googlebot.com or geo-crawl-.geo.googlebot.com and forward-confirm, or match Google's IP list.
Official documentation	https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

Should you block GoogleOther?

Its purpose is not clearly documented. Check your server logs to see how often it visits and what it requests before deciding.

Google says rules for GoogleOther do not affect any specific product, Search included. GoogleOther-Image and GoogleOther-Video also exist.

How to block or allow GoogleOther in robots.txt

Block it everywhere:

User-agent: GoogleOther
Disallow: /

Allow it everywhere:

User-agent: GoogleOther
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

Test GoogleOther on your robots.txt

How to let GoogleOther through a firewall or CDN

robots.txt only tells GoogleOther what it may fetch. If a firewall, a CDN or a bot-protection rule blocks the request first, it never reads the file, so allowing it in robots.txt is not enough.

Allow the IP ranges Google publishes (https://developers.google.com/static/crawling/ipranges/common-crawlers.json) rather than the user agent, which anyone can fake.

If your firewall can match hostnames, allow requests whose reverse DNS ends in .googlebot.com and resolves back to the same IP.

How to verify that a visit is really GoogleOther

Anyone can write GoogleOther in a user agent, so the user agent alone proves nothing. Take the IP address of the request from your server logs and check it.

Look up the hostname of the IP, then look up that hostname. The first answer must end in .googlebot.com, and the second must return the same IP; anything else is an impostor. On Windows, use nslookup instead of host.

host 203.0.113.7
host <hostname>

Google publishes its IP ranges as JSON. This command downloads the list and prints True if the IP is in it. Replace 203.0.113.7 with the IP from your logs; Python 3 comes with macOS and most Linux systems.

curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
  | python3 -c 'import sys, json, ipaddress as ip
a = ip.ip_address(sys.argv[1])
ranges = json.load(sys.stdin)["prefixes"]
print(any(a in ip.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges))' 203.0.113.7

Other Google crawlers

Googlebot: Search engine
Google-Extended: robots.txt control token
Google-CloudVertexBot: Other or undeclared

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.

What our data says about GoogleOther

Figures we measured on real AI answers and websites. Each one links to its study, with the method, the sample and the date.

5%

5% of the websites AI cites most in Spain block GoogleOther from crawling their site, and 4.7% name it in a robots.txt group of its own.

Sample: 678 websites AI cites for three or more of 43 brands in Spain. Source: Mencoro (November 2026), in Which AI crawlers do the websites AI cites let in?