Learning Center · Answers · Getting started
What's the difference between training crawlers, AI search crawlers and assistant fetchers?
Short answer
Training crawlers collect pages to train AI models, AI search crawlers index pages so AI search can cite and link to you, and assistant fetchers fetch a page in real time because a person asked about it. Block training crawlers by name if you choose. Allow the other two in robots.txt and at your CDN.
| Training crawlers | AI search crawlers | Assistant fetchers | |
|---|---|---|---|
| Why it visits | To collect pages for training AI models | To index pages so AI search can cite them | A person asked an assistant about a page |
| Examples | GPTBot, ClaudeBot, CCBot | OAI-SearchBot, Claude-SearchBot, PerplexityBot | ChatGPT-User, Claude-User, Perplexity-User |
| How it reads | Raw HTML, in bulk | Mostly raw HTML | Usually raw HTML, one page at a time |
| Value to you | Indirect at best | Visibility in AI answers | A customer asking about you right now |
| Sensible default | Your choice; block by name if you prefer | Allow | Allow |
| How to control it | robots.txt | robots.txt, plus CDN or bot protection | CDN or server rule; robots.txt may not apply |
In more detail
Each kind does a different job for a different company and brings you different value. A training crawler takes your content and may never send anyone back. An AI search crawler decides whether AI answers can cite and link to your pages. An assistant fetcher stands in for a customer asking about you right now, so these are usually the visits you want most.
All three mostly read raw HTML, so your key facts need to be in the page your server sends. Each uses its own robots.txt name, which lets you opt out of training and still appear in AI search and assistant answers. Some operators treat assistant fetches like a person clicking a link and say robots.txt may not apply, so stopping them completely takes a rule at your CDN or server.
What to do
- Decide whether your content may be used for AI training, as a business and legal choice.
- If you opt out, block training crawlers by name in robots.txt and leave the others allowed.
- Check your CDN and bot protection let AI search crawlers and assistant fetchers through.
- Put prices, stock, shipping and returns in the HTML your server sends.
Read more
How does your own site do?
AgentScore checks your site in about a minute and ranks what to fix first.