AI crawler index

What is meta-webindexer?

meta-webindexer is Meta's crawler for AI search. Navigates the web to improve the quality of Meta AI's search results for its users.

At a glance

Operator	Meta
Purpose	AI search
User agent	`meta-webindexer/1.1`
Obeys robots.txt	Yes, according to the operator
Published IP ranges	Not documented by the operator
How to verify it	Not documented by the operator
Official documentation	https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers

Should you block meta-webindexer?

It builds the index an AI search engine cites from. Blocking it can remove your pages from that engine's answers and links, so if you want to be cited there, allow it.

Meta says allowing it in robots.txt helps Meta AI cite and link your content in its answers. It is separate from meta-externalagent, which crawls for uses such as AI training. Meta may cache robots.txt for up to 24 hours.

How to block or allow meta-webindexer in robots.txt

Block it everywhere:

User-agent: meta-webindexer
Disallow: /

Allow it everywhere:

User-agent: meta-webindexer
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

How to verify that a visit is really meta-webindexer

Meta publishes no IP list or hostname for meta-webindexer, so you cannot prove a visit is genuine. Treat the user agent as a claim and judge the visits by what they request in your logs.

Other Meta crawlers

meta-externalagent: AI training
meta-externalfetcher: User-triggered fetch
facebookexternalhit: Link preview

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.