Agent directory · By company

Ai2's AI agents and crawlers

Ai2 runs 3 agents that visit websites: 1 AI assistant, 2 AI training crawlers. Here's what each one does, and how to let them in or keep them out.

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

AI2Bot-DeepResearchEval

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

AI2Bot: The Allen Institute for AI's crawler, which collects data for open research models and datasets.
Ai2Bot-Dolma

Robots.txt for every Ai2 agent

To keep Ai2's training crawlers out but still let its assistants, search and agents in:

User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
Disallow: /

To block every Ai2 agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Ai2

Our registry doesn't verify Ai2's agents against published IP ranges, so treat a request that uses one of their names as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.

Which Ai2 agents visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from them: what they read, what your site sent back, and whether each one was genuine. See agent traffic, or check whether AI agents can use your site for free.