Ghost Agent Lab Free AgentScore Start 30-day trial

Agent directory ยท By company

OpenAI's AI agents and crawlers

OpenAI runs 5 agents that visit websites: 1 AI assistant, 1 AI and search crawler, 1 autonomous agent, 2 AI training crawlers. Here's what each one does, and how to let them in or keep them out.

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

  • ChatGPT-User: Fetches a page when someone using ChatGPT, or a custom GPT, asks about it or clicks through to it. It acts for one person, in real time, so it's the closest thing to a customer visiting through ChatGPT.

Indexes pages so a search engine or AI search tool can find, cite and link to them. Allow it, unless you deliberately want to stay out of that search engine. Blocking it removes you from its results and answers.

  • OAI-SearchBot: Crawls the web so ChatGPT can show and link to sites in its search results. OpenAI says it isn't used to train its models.

Autonomous agents

Uses a website like a person would, clicking, typing and filling in forms to finish a task for someone. Make your journeys work for it. Many browser agents look like an ordinary browser, so you can't reliably block them by name; test whether they can finish your key tasks instead.

  • ChatGPT Agent: OpenAI's ChatGPT agent: a browser ChatGPT uses to carry out tasks for a user, such as researching, filling in forms or shopping.

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

  • GPTBot: Collects public web pages that may be used to train OpenAI's models. Blocking it doesn't stop your site appearing in ChatGPT search, which uses OAI-SearchBot.
  • OAI-AdsBot

Robots.txt for every OpenAI agent

To keep OpenAI's training crawlers out but still let its assistants, search and agents in:

User-agent: GPTBot
User-agent: OAI-AdsBot
Disallow: /

To block every OpenAI agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: OAI-AdsBot
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really OpenAI

OpenAI publishes ways to check that a request really came from it:

Requests that use OpenAI's names but fail these checks are impostors. Ghost Agent Lab checks every request this way. How to verify AI crawlers.

Which OpenAI agents visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from them: what they read, what your site sent back, and whether each one was genuine. See agent traffic, or check whether AI agents can use your site for free.