Learning Center · Answers · Access
What's the difference between ClaudeBot, Claude-User and Claude-SearchBot?
Short answer
ClaudeBot, Claude-SearchBot and Claude-User are Anthropic agents, each with its own robots.txt name. ClaudeBot collects public content that may be used to train Anthropic's models; Claude-SearchBot crawls to improve Claude's search results; Claude-User fetches a page when someone using Claude asks about it. Block ClaudeBot if you choose, and allow the other two.
| ClaudeBot | Claude-SearchBot | Claude-User | |
|---|---|---|---|
| Type | AI training crawler | AI search crawler | Assistant fetcher |
| What it does | Collects public content that may train models | Crawls to improve Claude's search results | Fetches a page when a Claude user asks |
| Should you allow it? | Your business choice | Allow, unless you want out of its results | Almost always allow |
| If you block it | AI assistants can still visit for a person | You drop out of its results and answers | Claude can't read your pages for a customer |
| Verifiable | No published IP ranges or reverse DNS | No published IP ranges or reverse DNS | No published IP ranges or reverse DNS |
In more detail
Because each name has its own robots.txt group, you can opt out of training without leaving Claude's search results or turning away customers who ask Claude about you. To keep only Anthropic's training crawlers out, list ClaudeBot and the older name anthropic-ai in one group with Disallow: /. A crawler follows only the most specific group that names it, so check your other groups too.
Anthropic doesn't publish IP ranges or a reverse DNS domain for these agents, so a request using one of their names can't be proven to come from Anthropic. Treat the name alone as a claim. Ghost Agent Lab reports these requests as unverified rather than guessing. Some agents that act for a person in real time, like Claude-User, may not read robots.txt.
What to do
- Decide whether Anthropic may use your content for training.
- To opt out, add a robots.txt group for ClaudeBot and anthropic-ai with Disallow: /.
- Check no wider rule blocks Claude-SearchBot or Claude-User.
- Treat requests using these names as unverified claims in your bot rules.
Read more
How does your own site do?
AgentScore checks your site in about a minute and ranks what to fix first.