Agent directory ยท By company
Common Crawl's AI agents and crawlers
Common Crawl runs one agent that visit websites: 1 AI training crawler. Here's what each one does, and how to let them in or keep them out.
AI training crawlers
Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.
- CCBot: Common Crawl's crawler. Common Crawl publishes a free archive of the web that many AI companies use as training data.
Robots.txt for every Common Crawl agent
To block every Common Crawl agent that reads robots.txt:
User-agent: CCBot
Disallow: /Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.
How to tell if it's really Common Crawl
Our registry doesn't verify Common Crawl's agents against published IP ranges, so treat a request that uses one of their names as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.
Which Common Crawl agents visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from them: what they read, what your site sent back, and whether each one was genuine. See agent traffic, or check whether AI agents can use your site for free.