Ghost Agent Lab Free AgentScore Start 30-day trial

Agent directory · Autonomous agent

Claude-Web

An older name linked to Anthropic. Its current agents are ClaudeBot, Claude-SearchBot and Claude-User; sites that block Anthropic often list this name in robots.txt too.

Run by
Anthropic
Type
Autonomous agent
Robots.txt name
Claude-Web
Verifiable
Not yet

What Claude-Web does

An older name linked to Anthropic. Its current agents are ClaudeBot, Claude-SearchBot and Claude-User; sites that block Anthropic often list this name in robots.txt too.

As an autonomous agent, it uses a website like a person would, clicking, typing and filling in forms to finish a task for someone. The kinds of AI agent that visit websites.

Should you allow it?

Make your journeys work for it. Many browser agents look like an ordinary browser, so you can't reliably block them by name; test whether they can finish your key tasks instead. A decision guide for blocking or allowing AI crawlers.

How to allow or block Claude-Web

Claude-Web reads the Claude-Web group in your robots.txt. To block it everywhere:

User-agent: Claude-Web
Disallow: /

To allow it everywhere, even if another rule blocks other bots:

User-agent: Claude-Web
Allow: /

Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.

How to tell if it's really Claude-Web

Our registry doesn't verify Claude-Web against published IP ranges yet, so treat a request that uses the name as a claim, not proof: anyone can send its user agent. Check Anthropic's documentation for how to verify it.

Every Anthropic agent, and a robots.txt for all of them →

Does Claude-Web visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from Claude-Web and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.