Which AI crawlers do the websites AI cites let in?

Key findings

1. 22.5% of the websites AI cites most in Spain block at least one AI crawler in their robots.txt.
2. 14.9% of these websites block GPTBot, which collects training data for OpenAI, but only 6.5% block OAI-SearchBot, which ChatGPT search uses.
3. The crawler these websites block most is Bytespider (ByteDance): 17.3% of them shut it out.
4. 21.5% of the websites AI cites most in Spain block at least one AI training crawler, against 14.1% that block an AI search crawler.
5. 11.5% of these websites block a crawler that fetches pages when a user asks, such as ChatGPT-User or Claude-User.
6. 10.2% of these websites opt out of Google’s AI training with Google-Extended, while 0.8% block Googlebot.
7. 21.3% of the robots.txt files of the websites AI cites in Spain have a line crawlers will not read as intended, most often a directive Google ignores, such as Crawl-delay.
8. 32.1% of these robots.txt files have a group of rules for at least one AI crawler.
9. 72.9% of these robots.txt files point crawlers to a sitemap.
10. 15.2% of the websites AI cites most in Spain publish an llms.txt file.
11. Of the 382 websites AI cites for three or more Spanish brands, 58 have an llms.txt and 86 block an AI crawler.

How many of these websites block each crawler

Blocked: the robots.txt stops the crawler from fetching the homepage, which in practice shuts it out of the whole site. Own group: the file names the crawler in a group of rules of its own, whatever those rules say.

Download the table (CSV)

Methodology

• Websites: every third-party domain cited in the AI answers of three or more of the 43 brands Mencoro tracks in Spain from July 8, 2026 to October 5, 2026, in ChatGPT, Google AI Overviews, Google AI Mode and Perplexity: 443 domains. The brands’ own domains are left out, and no brand is named.
• robots.txt: read on October 6, 2026 at https://domain/robots.txt (and www.), following redirects. A 404 counts as no rules. 61 domains could not be read (timeouts, server errors, or a bot shield answering 401 or 403) and are not counted: the percentages are over the 382 that could.
• Blocked: decided with the same RFC 9309 matcher as our robots.txt tester: the most specific user-agent group wins, then the longest matching rule. A crawler counts as blocked when that verdict stops it at the homepage.
• llms.txt: a file at /llms.txt that is plain text starting with a Markdown title, as the proposal describes. HTML pages served at that address do not count.
• Limits: robots.txt is what a site asks for, not what its firewall enforces, and crawlers that fetch on a user’s behalf may not read it. The websites are the ones cited for Spanish commercial questions, not a random sample of the web.

How to cite this study

Mencoro, “Which AI crawlers do the websites AI cites let in?”, October 2026, https://mencoro.com/data-studies/ai-crawler-access/. You are welcome to use the figures with a link to this page.