AI search crawler

A bot that indexes pages so AI search and assistants can cite and link to them, such as OAI-SearchBot or PerplexityBot.

Why it matters

AI search crawlers decide whether your pages can be cited and linked in AI answers, which is where a growing share of product research now starts. If they're blocked, or can't read your pages, assistants recommend competitors they could read instead. Most sites should let them in, and check now and then that robots.txt and bot protection still do.

More terms in the agents

• AI agent Software that uses an AI model to do something on the web for a person or a company: read a page, answer a question, compare products or complete a task.
• Training crawler A bot that collects pages to train AI models, such as GPTBot or ClaudeBot.
• Assistant fetcher An AI assistant fetching a page in real time because a person asked it something, such as ChatGPT-User or Claude-User.
• Browser agent An agent that opens your site in a real browser and clicks, types and scrolls like a person, to book, sign up or buy.
• User agent The name a visitor sends with every request to say what it is, such as a browser version or GPTBot.

Every term, A to Z → · Read the glossary in full →