Ghost Agent Lab Free AgentScore Start 30-day trial

Agent directory · AI training crawler

anthropic-ai

An older robots.txt name linked to Anthropic's crawling. Its current crawler is ClaudeBot; sites that block Anthropic's training often list this name too.

Run by
Anthropic
Type
AI training crawler
Robots.txt name
anthropic-ai
Verifiable
Not yet

What anthropic-ai does

An older robots.txt name linked to Anthropic's crawling. Its current crawler is ClaudeBot; sites that block Anthropic's training often list this name too.

As an AI training crawler, it collects public pages that may be used to train AI models. The kinds of AI agent that visit websites.

Should you allow it?

A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else. A decision guide for blocking or allowing AI crawlers.

How to allow or block anthropic-ai

anthropic-ai reads the anthropic-ai group in your robots.txt. To block it everywhere:

User-agent: anthropic-ai
Disallow: /

To allow it everywhere, even if another rule blocks other bots:

User-agent: anthropic-ai
Allow: /

Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.

How to tell if it's really anthropic-ai

Our registry doesn't verify anthropic-ai against published IP ranges yet, so treat a request that uses the name as a claim, not proof: anyone can send its user agent. Check Anthropic's documentation for how to verify it.

Every Anthropic agent, and a robots.txt for all of them →

Does anthropic-ai visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from anthropic-ai and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.