AI crawler index
What is YouBot?
YouBot is You.com's crawler for AI search. Crawls and indexes pages to power the You.com search engine.
At a glance
| Operator | You.com |
|---|---|
| Purpose | AI search |
| User agent | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36 |
| Obeys robots.txt | Yes, according to the operator |
| Published IP ranges | Not documented by the operator |
| How to verify it | Web Bot Auth signatures, reverse DNS to youbot-*.search.you.com, or the IP range in You.com's docs. |
| Official documentation | https://you.com/docs/youbot |
Should you block YouBot?
It builds the index an AI search engine cites from. Blocking it can remove your pages from that engine's answers and links, so if you want to be cited there, allow it.
It honors robots.txt, including Crawl-delay, and caches it for 30 minutes. You.com does not state whether the data trains AI models.
How to block or allow YouBot in robots.txt
Block it everywhere:
User-agent: YouBot
Disallow: /Allow it everywhere:
User-agent: YouBot
Allow: /Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.
Check your own site
The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.
FAQ
YouBot questions
Start tracking your brand in AI search today
Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.