Learning Center · Answers · Access

What's the difference between ClaudeBot, Claude-User and Claude-SearchBot?

Short answer

ClaudeBot, Claude-SearchBot and Claude-User are Anthropic agents, each with its own robots.txt name. ClaudeBot collects public content that may be used to train Anthropic's models; Claude-SearchBot crawls to improve Claude's search results; Claude-User fetches a page when someone using Claude asks about it. Block ClaudeBot if you choose, and allow the other two.

Comparison · Last checked November 12, 2026 · Based on Anthropic's AI agents and crawlers, robots.txt for AI agents: who to allow and who to block

What's the difference between ClaudeBot, Claude-User and Claude-SearchBot?
ClaudeBotClaude-SearchBotClaude-User
TypeAI training crawlerAI search crawlerAssistant fetcher
What it doesCollects public content that may train modelsCrawls to improve Claude's search resultsFetches a page when a Claude user asks
Should you allow it?Your business choiceAllow, unless you want out of its resultsAlmost always allow
If you block itAI assistants can still visit for a personYou drop out of its results and answersClaude can't read your pages for a customer
VerifiableNo published IP ranges or reverse DNSNo published IP ranges or reverse DNSNo published IP ranges or reverse DNS

In more detail

Because each name has its own robots.txt group, you can opt out of training without leaving Claude's search results or turning away customers who ask Claude about you. To keep only Anthropic's training crawlers out, list ClaudeBot and the older name anthropic-ai in one group with Disallow: /. A crawler follows only the most specific group that names it, so check your other groups too.

Anthropic doesn't publish IP ranges or a reverse DNS domain for these agents, so a request using one of their names can't be proven to come from Anthropic. Treat the name alone as a claim. Ghost Agent Lab reports these requests as unverified rather than guessing. Some agents that act for a person in real time, like Claude-User, may not read robots.txt.

What to do

  1. Decide whether Anthropic may use your content for training.
  2. To opt out, add a robots.txt group for ClaudeBot and anthropic-ai with Disallow: /.
  3. Check no wider rule blocks Claude-SearchBot or Claude-User.
  4. Treat requests using these names as unverified claims in your bot rules.

Read more

All answers

How does your own site do?

AgentScore checks your site in about a minute and ranks what to fix first.