robots.txt tester for AI agents
See which AI assistants, search crawlers and training crawlers your robots.txt lets in, and the exact rule that decides each, read the same way AgentScore reads it.
Questions
Which agents does it check?
Every AI assistant, AI search crawler, AI training crawler and autonomous agent in our agent directory, plus the big search engines. Each links to a page about what it does and whether to allow it.
Does it follow the same rules as AgentScore?
Yes. It reads the file the way AgentScore does: each agent follows the group with its own name, or the * group if there isn't one; of the rules that match the page, the longest wins, and Allow wins a tie; * matches anything and $ marks the end of the address. AgentScore's robots.txt check looks at your home page, and fails when three or more AI assistants and search agents are blocked from it.
Should I block AI training crawlers?
It's a business choice. If your content is the product (publishing, research, courses), blocking training crawlers like GPTBot and CCBot can make sense. If your pages exist to sell something else, being in AI models usually helps. Either way, keep AI assistants and AI search crawlers allowed: they're how people find and buy from you through AI.
Does blocking in robots.txt actually stop them?
Only the agents that respect it. The major AI companies say their crawlers do. Impostors and scrapers don't. To see who actually visits and whether they follow your rules, you need your server logs: that's what Ghost Agent Lab's agent traffic shows.
Can I test a page other than the home page?
Yes. Change the page to any path on the site, like /products/ or /cart, and the results update as you type.
See which agents actually visit
robots.txt says who's welcome. Ghost Agent Lab shows who really comes, whether they follow your rules, and what they do on your site.