Meta's AI agents and crawlers

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

• Meta-ExternalFetcher: Fetches a page when someone using Meta AI, or another Meta product, asks for it, so it acts for a person in real time.
• meta-webindexer

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

• Meta-ExternalAgent: Meta's crawler for collecting web content that may be used to train its AI models and improve its products.
• FacebookBot

Other bots

A well-known bot that isn't an AI agent, such as an ads checker or a link preview. Usually allow it: these bots support products you already use.

• facebookexternalhit: Fetches a page to build the preview when someone shares a link on Facebook, Instagram, Messenger or WhatsApp.

Robots.txt for every Meta agent

To keep Meta's training crawlers out but still let its assistants, search and agents in:

User-agent: meta-externalagent
User-agent: FacebookBot
Disallow: /

To block every Meta agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: facebookexternalhit
User-agent: FacebookBot
User-agent: meta-webindexer
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Meta

Our registry doesn't verify Meta's agents against published IP ranges, so treat a request that uses one of their names as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.