AI crawler index
What is Google-Extended?
Google-Extended is a robots.txt control token from Google, not a crawler. Controls whether content Google crawls is used to train future Gemini models and to ground Gemini Apps and Vertex AI.
At a glance
| Operator | |
|---|---|
| Purpose | robots.txt control token |
| User agent | Not documented by the operator |
| Obeys robots.txt | Yes, according to the operator |
| Published IP ranges | Not documented by the operator |
| How to verify it | Not documented by the operator |
| Official documentation | https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers |
Should you block Google-Extended?
It has no crawler of its own: it is a robots.txt switch that controls how data collected by the operator's main crawler is used. Blocking it does not stop that crawler or remove you from its search results.
It has no crawler of its own: Google uses it only as a robots.txt control. Google says blocking it does not affect inclusion in Google Search and is not a ranking signal.
How to block or allow Google-Extended in robots.txt
Block it everywhere:
User-agent: Google-Extended
Disallow: /Allow it everywhere:
User-agent: Google-Extended
Allow: /Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.
Other Google crawlers
- Googlebot: Search engine
- GoogleOther: Other or undeclared
Check your own site
The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.
FAQ
Google-Extended questions
Start tracking your brand in AI search today
Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.