Learning Center · Explain it to my boss · robots.txt, llms.txt and AI crawlers
Blocking the wrong AI crawlers can remove us from AI answers
A short file on our website, robots.txt, tells AI companies' crawlers what they may read. A careless rule, or a copied list that blocks all AI, can shut out the assistants and AI search that answer customers' questions about us, not just the crawlers that train AI models. Then we're left out of answers and recommendations.
What it costs the business
- If robots.txt shuts out AI assistants and AI search, they can't read our pages and leave us out of answers and recommendations.
- Long block-all-AI lists usually include assistants and AI search too, removing us from AI answers as well as training.
- Sites are often launched with a leftover rule that blocks everything, hiding the whole site from agents.
What to do
- Decide on AI training separately: blocking training crawlers is a legitimate policy choice, made once with legal.
- Allow AI search crawlers and assistant fetchers, which answer customers' questions about us and link back.
- Remove any leftover rule that blocks everything, and keep product and pricing pages open to agents.
- Check bot protection too: a firewall rule that challenges AI crawlers overrides a friendly robots.txt.
Who owns it: SEO or content teamEffort: Hours
The detail is in robots.txt, llms.txt and AI crawlers. Based on robots.txt for AI agents: who to allow and who to block, Should you block AI crawlers? A decision guide for leaders, How to explain agent readiness to your board.