Cohere's AI agents and crawlers

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

• cohere-ai

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

• cohere-training-data-crawler: Cohere's crawler for collecting data that may be used to train its models.

Robots.txt for every Cohere agent

To keep Cohere's training crawlers out but still let its assistants, search and agents in:

User-agent: cohere-training-data-crawler
Disallow: /

To block every Cohere agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: cohere-training-data-crawler
User-agent: cohere-ai
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Cohere

Our registry doesn't verify Cohere's agents against published IP ranges, so treat a request that uses one of their names as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.