Learning Center · Answers · Access

Should I allow AI training crawlers?

Short answer

It's a real business choice. Lean towards allowing training crawlers if your pages mainly exist to sell something, such as product pages, help centers and pricing. Lean towards blocking if your content is the product, such as publishing, research or courses. Either way, block them by name so AI search and assistants can still reach you.

Answer · Last checked November 12, 2026 · Based on Should you block AI crawlers? A decision guide for leaders, robots.txt for AI agents: who to allow and who to block

In more detail

Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train future AI models. Allowing them means your content may shape what AI models know, including about your brand and products. Blocking them keeps your content out of future training sets and leaves AI search and assistants unaffected, because those use different names. Google-Extended and Applebot-Extended are control names that say whether Google and Apple may use what they crawl for their AI models.

For an online store the usual answer is to allow, because product pages are there to be read and being well understood by AI models is usually worth more than the content itself. Software and services companies usually allow marketing pages and docs. A publisher or content business often blocks, or allows only under a licensing agreement. If you're unsure, decide deliberately rather than inheriting someone's copied list.

The costly mistake is blocking too much. Long "block all AI" lists usually include assistant fetchers and AI search crawlers, which removes you from AI answers as well as training. AgentScore reports blocked training crawlers without taking points off, because either answer is legitimate. Write the decision down, sign it off with marketing, e-commerce and whoever owns your content rights, and revisit it every quarter.

What to do

  1. Decide whether your content is the product or exists to sell something else.
  2. Get sign-off from marketing, e-commerce and your content rights owner.
  3. If blocking, list only training names such as GPTBot, ClaudeBot, CCBot and Google-Extended.
  4. Check that AI search crawlers and assistant fetchers stay allowed.
  5. Make sure your CDN's AI settings match the policy.

Read more

All answers

How does your own site do?

AgentScore checks your site in about a minute and ranks what to fix first.