AI crawler index

What is Google-CloudVertexBot?

Google-CloudVertexBot is a crawler run by Google. Crawls sites whose owners ask for it while building Vertex AI Agents in Google Cloud.

At a glance

Operator	Google
Purpose	Other or undeclared
User agent	Not documented by the operator
Obeys robots.txt	Yes, according to the operator
Published IP ranges	https://developers.google.com/static/crawling/ipranges/common-crawlers.json
How to verify it	Reverse DNS to crawl-.googlebot.com or geo-crawl-.geo.googlebot.com and forward-confirm, or match Google's IP list.
Official documentation	https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers

Should you block Google-CloudVertexBot?

Google says it crawls only when the owner of the site asks for it, to build a Vertex AI Agent, and that blocking it has no effect on Google Search. Block it only if you do not want your own agent builds to read the site.

Google documents only the user agent substring "Google-CloudVertexBot", not a full string. Rules for it have no effect on Google Search or other Google products.

How to block or allow Google-CloudVertexBot in robots.txt

Block it everywhere:

User-agent: Google-CloudVertexBot
Disallow: /

Allow it everywhere:

User-agent: Google-CloudVertexBot
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

How to verify that a visit is really Google-CloudVertexBot

Anyone can write Google-CloudVertexBot in a user agent, so the user agent alone proves nothing. Take the IP address of the request from your server logs and check it.

Look up the hostname of the IP, then look up that hostname. The first answer must end in .googlebot.com, and the second must return the same IP; anything else is an impostor. On Windows, use nslookup instead of host.

host 203.0.113.7
host <hostname>

Google publishes its IP ranges as JSON. This command downloads the list and prints True if the IP is in it. Replace 203.0.113.7 with the IP from your logs; Python 3 comes with macOS and most Linux systems.

curl -s https://developers.google.com/static/crawling/ipranges/common-crawlers.json \
  | python3 -c 'import sys, json, ipaddress as ip
a = ip.ip_address(sys.argv[1])
ranges = json.load(sys.stdin)["prefixes"]
print(any(a in ip.ip_network(r.get("ipv4Prefix") or r.get("ipv6Prefix")) for r in ranges))' 203.0.113.7

Other Google crawlers

Googlebot: Search engine
Google-Extended: robots.txt control token
GoogleOther: Other or undeclared

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.