Agent directory ยท AI training crawler
AI2Bot
The Allen Institute for AI's crawler, which collects data for open research models and datasets.
- Run by
- Ai2
- Type
- AI training crawler
- Robots.txt name
AI2Bot- Verifiable
- Not yet
What AI2Bot does
The Allen Institute for AI's crawler, which collects data for open research models and datasets.
As an AI training crawler, it collects public pages that may be used to train AI models. The kinds of AI agent that visit websites.
Should you allow it?
A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else. A decision guide for blocking or allowing AI crawlers.
How to allow or block AI2Bot
AI2Bot reads the AI2Bot group in your robots.txt. To block it everywhere:
User-agent: AI2Bot
Disallow: /To allow it everywhere, even if another rule blocks other bots:
User-agent: AI2Bot
Allow: /Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.
How to tell if it's really AI2Bot
Our registry doesn't verify AI2Bot against published IP ranges yet, so treat a request that uses the name as a claim, not proof: anyone can send its user agent. Check Ai2's documentation for how to verify it.
Does AI2Bot visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from AI2Bot and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.