Ghost Agent Lab Free AgentScore Start 30-day trial

Agent directory · AI training crawler

GPTBot

Collects public web pages that may be used to train OpenAI's models. Blocking it doesn't stop your site appearing in ChatGPT search, which uses OAI-SearchBot.

Run by
OpenAI
Type
AI training crawler
Robots.txt name
GPTBot
Verifiable
Yes

What GPTBot does

Collects public web pages that may be used to train OpenAI's models. Blocking it doesn't stop your site appearing in ChatGPT search, which uses OAI-SearchBot.

As an AI training crawler, it collects public pages that may be used to train AI models. The kinds of AI agent that visit websites.

Should you allow it?

A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else. A decision guide for blocking or allowing AI crawlers.

How to allow or block GPTBot

GPTBot reads the GPTBot group in your robots.txt. To block it everywhere:

User-agent: GPTBot
Disallow: /

To allow it everywhere, even if another rule blocks other bots:

User-agent: GPTBot
Allow: /

Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.

How it identifies itself

A typical user agent:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot

How to tell if it's really GPTBot

You can check a request that claims to be GPTBot against what OpenAI publishes:

Requests that use the name but fail these checks are impostors, usually scrapers trying to get past bot protection. Ghost Agent Lab checks every request this way and gives you rules to block impostors. How to verify AI crawlers.

Every OpenAI agent, and a robots.txt for all of them →

Does GPTBot visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from GPTBot and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.