AI crawler index

What is meta-externalfetcher?

meta-externalfetcher is Meta's fetcher for pages users ask about. Fetches individual links at a user's request and supports agentic AI that navigates sites for users.

At a glance

Operator	Meta
Purpose	User-triggered fetch
User agent	`meta-externalfetcher/1.1`
Obeys robots.txt	Not always: the operator says user-requested visits may ignore it
Published IP ranges	Not documented by the operator
How to verify it	Not documented by the operator
Official documentation	https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers

Should you block meta-externalfetcher?

It visits a page only when a user asks the assistant something that needs it. Blocking it stops the assistant from reading your page in those moments, which costs you citations; and because a person starts each request, robots.txt may not stop it.

Meta says it may bypass robots.txt because the fetch is user-requested. It is separate from facebookexternalhit, which builds link previews.

How to block or allow meta-externalfetcher in robots.txt

Block it everywhere:

User-agent: meta-externalfetcher
Disallow: /

Allow it everywhere:

User-agent: meta-externalfetcher
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

Other Meta crawlers

meta-externalagent: AI training

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.

FAQ

meta-externalfetcher questions

meta-externalfetcher is Meta's fetcher for pages users ask about. Fetches individual links at a user's request and supports agentic AI that navigates sites for users.

Not always. Meta says that, because a user starts each visit, robots.txt rules may not apply. A firewall or IP rule is the reliable way to stop it.

Add "User-agent: meta-externalfetcher" followed by "Disallow: /" to your robots.txt. Because robots.txt may not apply to it, back the rule with a firewall or IP rule. Meta says it may bypass robots.txt because the fetch is user-requested. It is separate from facebookexternalhit, which builds link previews.

Start tracking your brand in AI search today

Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.