Learning Center · Answers · Access
Will AI agents respect my robots.txt?
Short answer
Well-behaved AI crawlers follow robots.txt, and scrapers ignore it. Some operators say it may not apply when an assistant fetches a page because a person asked, and browser agents look like ordinary browsers, so rules by name don't reach them. Use robots.txt to state your policy, and enforce it at your CDN by verifying who's asking.
In more detail
robots.txt is a request. It asks crawlers to stay out of certain pages, and the major AI companies give each of their agents its own name so you can address them one by one. Their training crawlers and AI search crawlers read their own group and follow it. A crawler follows only the most specific group that names it, so a group for one agent replaces your general rules for that agent.
Three gaps remain. Scrapers ignore robots.txt, and many borrow trusted names like GPTBot. Some operators treat an assistant fetch, made because a person asked about a specific page, like a person clicking a link, and say robots.txt may not apply. Browser agents mostly look like a normal browser, so rules by name don't reach them.
That's why enforcement belongs at your CDN or firewall. Stop abusive traffic there by checking where requests really come from, and keep your CDN settings in line with robots.txt. A CDN rule that blocks or challenges an agent overrides a friendly robots.txt, because the agent never gets far enough to read it.
What to do
- Write one policy per kind of agent and put it in robots.txt.
- Check each agent you want follows a group that allows your important pages.
- Ask your CDN admin to stop impostors by verifying identity, never by trusting names.
- Confirm your CDN and bot protection settings match robots.txt.
- Test whether browser agents can complete a purchase or sign-up.
Read more
How does your own site do?
AgentScore checks your site in about a minute and ranks what to fix first.