The AI crawler checklist
Everything you need to decide which AI agents to let into your website, on one printable page. Free, for your email address.
What's inside
- Who runs what. The robots.txt names of 56 AI agents from 25 companies, including Google, Anthropic, Amazon, OpenAI, Mistral AI, sorted into assistants, search, autonomous agents and training crawlers.
- What to allow. A plain recommendation for each kind of agent, and which ones you can prove are real.
- 8 things to check. The same checks AgentScore runs, from robots.txt and bot protection to content that loads without JavaScript.
- A robots.txt to start from. Let AI assistants and search in, and keep training crawlers out if you choose to.
Questions
What's in it?
One page: which kinds of AI agent to let in, the robots.txt names each major AI company uses (and which you can verify), the things to check on your site, and a robots.txt to start from.
Is it up to date?
It's built from our AI agent directory and AgentScore's checks, so it changes when they do. The date in the corner is the directory version it was made from.
What will you email me?
New research, guides and news about Ghost Agent Lab, when we publish them. Unsubscribe anytime from any email. See the privacy policy.
Check your site against it, automatically
AgentScore runs the same checks on your site in about a minute, and ranks what to fix first.