Apple's AI agents and crawlers

AI and search crawlers

Indexes pages so a search engine or AI search tool can find, cite and link to them. Allow it, unless you deliberately want to stay out of that search engine. Blocking it removes you from its results and answers.

• Applebot: Apple's crawler for Siri, Spotlight and Safari suggestions.

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

• Applebot-Extended: Not a crawler: a robots.txt name Apple reads to decide whether content Applebot collects may be used to train Apple's generative AI models. Blocking it doesn't remove you from Siri or Spotlight.

Robots.txt for every Apple agent

To keep Apple's training crawlers out but still let its assistants, search and agents in:

User-agent: Applebot-Extended
Disallow: /

To block every Apple agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: Applebot
User-agent: Applebot-Extended
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Apple

Apple publishes ways to check that a request really came from it:

• IP ranges: search.developer.apple.com/applebot.json
• Reverse DNS: names ending in applebot.apple.com

Requests that use Apple's names but fail these checks are impostors. Ghost Agent Lab checks every request this way. How to verify AI crawlers.