AI crawler index

What is meta-externalfetcher?

meta-externalfetcher is Meta's fetcher for pages users ask about. Fetches individual links at a user's request and supports agentic AI that navigates sites for users.

At a glance

OperatorMeta
PurposeUser-triggered fetch
User agent meta-externalfetcher/1.1
Obeys robots.txtNot always: the operator says some visits may ignore it
Published IP ranges Not documented by the operator
How to verify it Not documented by the operator
Official documentation https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers

Should you block meta-externalfetcher?

It visits a page only when a user asks the assistant something that needs it. Blocking it stops the assistant from reading your page in those moments, which costs you citations; and because a person starts each request, robots.txt may not stop it.

Meta says it may bypass robots.txt because the fetch is user-requested. It is separate from facebookexternalhit, which builds link previews.

How to block or allow meta-externalfetcher in robots.txt

Block it everywhere:

User-agent: meta-externalfetcher
Disallow: /

Allow it everywhere:

User-agent: meta-externalfetcher
Allow: /

Rules are matched by the token on the User-agent line. Test the result with the robots.txt tester before you publish it.

Test meta-externalfetcher on your robots.txt

How to let meta-externalfetcher through a firewall or CDN

robots.txt only tells meta-externalfetcher what it may fetch. If a firewall, a CDN or a bot-protection rule blocks the request first, it never reads the file, so allowing it in robots.txt is not enough.

Meta publishes no IP list or hostname for meta-externalfetcher, so the only rule you can write is on the user agent, and that also lets in anyone who copies it.

How to verify that a visit is really meta-externalfetcher

Meta publishes no IP list or hostname for meta-externalfetcher, so you cannot prove a visit is genuine. Treat the user agent as a claim and judge the visits by what they request in your logs.

Other Meta crawlers

Check your own site

The AI crawler checker tells you which AI crawlers your robots.txt lets in, and the log file analyzer shows how often each one actually visits. For the bigger picture, see what AI crawlers are and the full AI crawler index.

What our data says about meta-externalfetcher

Figures we measured on real AI answers and websites. Each one links to its study, with the method, the sample and the date.

  • 3.5%

    3.5% of the websites AI cites most in Spain block meta-externalfetcher from crawling their site, and 3.4% name it in a robots.txt group of its own.
    Sample: 678 websites AI cites for three or more of 43 brands in Spain. Source: Mencoro (November 2026), in Which AI crawlers do the websites AI cites let in?

FAQ

meta-externalfetcher questions

meta-externalfetcher is Meta's fetcher for pages users ask about. Fetches individual links at a user's request and supports agentic AI that navigates sites for users.
Not always. Meta says robots.txt rules may not apply to some of its visits. A firewall or IP rule is the reliable way to stop it.
Add "User-agent: meta-externalfetcher" followed by "Disallow: /" to your robots.txt. Because robots.txt may not apply to it, back the rule with a firewall or IP rule. Meta says it may bypass robots.txt because the fetch is user-requested. It is separate from facebookexternalhit, which builds link previews.
Allow it in robots.txt with "User-agent: meta-externalfetcher" and "Allow: /", then make sure your firewall or CDN does not block it before it reaches the file.
Meta publishes no IP list or hostname for it, so only the user agent identifies it, and anyone can fake that.

Start tracking your brand in AI search today

Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.