Anthropic's AI agents and crawlers

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

• Claude-User: Fetches a page when someone using Claude asks about it or shares a link. It acts for one person, in real time.

AI and search crawlers

Indexes pages so a search engine or AI search tool can find, cite and link to them. Allow it, unless you deliberately want to stay out of that search engine. Blocking it removes you from its results and answers.

• Claude-SearchBot: Crawls the web to improve the search results Claude gives its users.

Autonomous agents

Uses a website like a person would, clicking, typing and filling in forms to finish a task for someone. Make your journeys work for it. Many browser agents look like an ordinary browser, so you can't reliably block them by name; test whether they can finish your key tasks instead.

• Claude-Code: Anthropic's coding agent. It fetches web pages, such as documentation, when a developer asks it to.
• Claude-Web: An older name linked to Anthropic. Its current agents are ClaudeBot, Claude-SearchBot and Claude-User; sites that block Anthropic often list this name in robots.txt too.

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

• ClaudeBot: Anthropic's crawler for collecting public web content that may be used to train its models.
• anthropic-ai: An older robots.txt name linked to Anthropic's crawling. Its current crawler is ClaudeBot; sites that block Anthropic's training often list this name too.

Robots.txt for every Anthropic agent

To keep Anthropic's training crawlers out but still let its assistants, search and agents in:

User-agent: ClaudeBot
User-agent: anthropic-ai
Disallow: /

To block every Anthropic agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Code
User-agent: Claude-Web
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Anthropic

Our registry doesn't verify Anthropic's agents against published IP ranges, so treat a request that uses one of their names as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.