Training crawler

A bot that collects pages to train AI models, such as GPTBot or ClaudeBot. Blocking it doesn't stop AI assistants visiting for a person.

Why it matters

Training crawlers are the agents most sites argue about, because they take content without sending visitors back. Blocking them is a reasonable business choice, but it's often done with rules broad enough to block the assistants that do bring customers. Decide on training crawlers separately, by name, so a choice about AI training doesn't quietly cost you AI search and shopping traffic.

More terms in the agents

• AI agent Software that uses an AI model to do something on the web for a person or a company: read a page, answer a question, compare products or complete a task.
• AI search crawler A bot that indexes pages so AI search and assistants can cite and link to them, such as OAI-SearchBot or PerplexityBot.
• Assistant fetcher An AI assistant fetching a page in real time because a person asked it something, such as ChatGPT-User or Claude-User.
• Browser agent An agent that opens your site in a real browser and clicks, types and scrolls like a person, to book, sign up or buy.
• User agent The name a visitor sends with every request to say what it is, such as a browser version or GPTBot.

Every term, A to Z → · Read the glossary in full →