AssistantBlocking it costs you visibility
Meta-ExternalFetcher
Opens a link a Meta AI user shared in a conversation.
- robots.txt token
meta-externalfetcher- Operator
- Meta
- Powers
- Meta AI live page reads
- Honours robots.txt
- Partially
Allow it
User-agent: meta-externalfetcher
Allow: /Block it
User-agent: meta-externalfetcher
Disallow: /Need the whole file, with every crawler and your private paths — and the llms.txt to go with it? The generator writes them. AI config generator →
User-Agent
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) Crawlable/1.0; +https://crawlable.fr/probeThis is the string we send when probing. The Crawlable suffix identifies us in your logs.
Meta
Check your own site
robots.txt is only half the answer: your CDN can refuse Meta-ExternalFetcher before it ever reads the file. A scan checks both.
Free, no account, about 5 seconds.