Learning Center · Answers · Access
Should I block GPTBot?
Short answer
Block GPTBot only if you don't want your content used to train OpenAI's models. It's a training crawler, so blocking it leaves ChatGPT search and ChatGPT's real-time page fetches untouched. Stores whose pages exist to sell products usually allow it. Publishers and others whose content is the product often block it.
In more detail
GPTBot collects public web pages that may be used to train OpenAI's models. OpenAI runs other agents under different names. OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a person asks about it. Each reads its own group in robots.txt, so you can say no to GPTBot and still say yes to the agents that answer customers' questions and link back to you.
Treat this as a business decision about your content. Lean towards blocking if your content is what you sell, such as publishing, research, courses or original reviews. Lean towards allowing if your pages exist to sell something else, like product pages, help centers and pricing. For those, being well understood by AI models is usually worth more than the content itself. AgentScore reports a blocked training crawler but doesn't take points off for it.
Whatever you decide, write it into robots.txt as its own group and check your CDN too. A copied "block all AI" list usually blocks assistants and AI search along with training. And robots.txt only asks. Well-behaved crawlers follow it, while a scraper can send the GPTBot name and ignore your rules, so stopping those has to happen at your CDN.
What to do
- Decide whether your content is the product or exists to sell something else.
- Agree a written policy with marketing, e-commerce and whoever owns your content rights.
- If you choose to block, add a GPTBot group with Disallow: / to robots.txt.
- Check that your rules still let OAI-SearchBot and ChatGPT-User in.
- Ask your CDN admin to confirm its AI settings match the policy.
Read more
How does your own site do?
AgentScore checks your site in about a minute and ranks what to fix first.