robots.txt, llms.txt and AI crawlers
Which AI crawlers you let in decides whether you show up in AI answers. Everything we have on robots.txt, llms.txt and choosing which assistants, search crawlers and training crawlers to allow.
Start hererobots.txt for AI agents: who to allow and who to blockThe three kinds of AI agent, the robots.txt names each one uses, a sensible default, and the most-specific-group rule that catches most people out.Read the guide
What AgentScore checks
- robots.txt lets AI assistants and search agents in If robots.txt shuts out AI assistants and AI search, they can't read your pages, so they leave you out of answers and recommendations. Usually fixed by SEO or content team
- llms.txt guide for AI An llms.txt file gives AI a quick, accurate summary of your business and key pages. It's new, so few competitors have one yet. Usually fixed by SEO or content team
- Agents can use your site through MCP or WebMCP MCP or WebMCP tools let agents search, check stock and build a cart directly, which is faster and more reliable than clicking through pages. A server description over 1 MB still counts: we read its start. Usually fixed by Developer
Guides
How to write an llms.txt fileWhat llms.txt is, what to put in it, a complete example for an online store, and the mistakes that make it useless to AI agents.
MCP and WebMCP: giving AI agents a direct way into your siteWhat MCP servers and WebMCP tools are, what they mean for your business, how AgentScore detects them, and how to start safely.
The AI agents that visit your website, and what each one wantsTraining crawlers, AI search crawlers, assistant fetchers, browser agents and tool-using agents: what each does, what it needs from your site, and how to treat it.
How to tell if an AI crawler is realAnyone can claim to be GPTBot or Googlebot. How to verify AI agents with published IP ranges, reverse DNS and signed requests, and what to do when there's no proof.
From the blog
Should you block AI crawlers? A decision guide for leadersBlocking AI crawlers is several decisions, not one. How to decide on training crawlers, AI search and assistants, and how to enforce the policy.
Should you build an MCP server yet?A decision guide for leaders: who should build an MCP server now, who should wait, what to build first, the risks and costs, and the cheaper fixes first.
Free tools
Terms to know
- robots.txt A file at the root of your site that tells crawlers and agents which pages they may visit.
- llms.txt A short Markdown file at /llms.txt that tells AI what your business does and links to your key pages.
- User agent The name a visitor sends with every request to say what it is, such as a browser version or GPTBot.
- Training crawler A bot that collects pages to train AI models, such as GPTBot or ClaudeBot.
- AI search crawler A bot that indexes pages so AI search and assistants can cite and link to them, such as OAI-SearchBot or PerplexityBot.
- Assistant fetcher An AI assistant fetching a page in real time because a person asked it something, such as ChatGPT-User or Claude-User.
- MCP (Model Context Protocol) A standard way to give AI agents direct tools for your site, such as search, product details and cart, instead of making them click through pages.
- WebMCP A proposal for offering those same tools from inside your web pages, so a browser agent can call them while it's on your site.