AI agent

Software that uses an AI model to do something on the web for a person or a company: read a page, answer a question, compare products or complete a task. In our guides, “agent” covers everything from crawlers to agents that shop.

Why it matters

AI agents are becoming a new way customers find, compare and buy. When someone asks an assistant for the best running shoe under $150, the answer depends on which sites its agents could read and use. A site agents can't use doesn't get a complaint or a support ticket; it simply isn't recommended. Knowing which agents visit, and whether they succeed, is the starting point.

More terms in the agents

• Training crawler A bot that collects pages to train AI models, such as GPTBot or ClaudeBot.
• AI search crawler A bot that indexes pages so AI search and assistants can cite and link to them, such as OAI-SearchBot or PerplexityBot.
• Assistant fetcher An AI assistant fetching a page in real time because a person asked it something, such as ChatGPT-User or Claude-User.
• Browser agent An agent that opens your site in a real browser and clicks, types and scrolls like a person, to book, sign up or buy.
• User agent The name a visitor sends with every request to say what it is, such as a browser version or GPTBot.

Every term, A to Z → · Read the glossary in full →