Agent directory · AI training crawler
Google-Extended
Not a crawler: a robots.txt name Google reads to decide whether content Googlebot collects may be used to train and ground Gemini models. Blocking it doesn't affect Google Search.
- Run by
- Type
- AI training crawler
- Robots.txt name
Google-Extended- Verifiable
- No
What Google-Extended does
Not a crawler: a robots.txt name Google reads to decide whether content Googlebot collects may be used to train and ground Gemini models. Blocking it doesn't affect Google Search.
As an AI training crawler, it collects public pages that may be used to train AI models. The kinds of AI agent that visit websites.
Should you allow it?
A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else. A decision guide for blocking or allowing AI crawlers.
How to allow or block Google-Extended
Google-Extended is only a name in robots.txt, so this is the only way to control it. To block it everywhere:
User-agent: Google-Extended
Disallow: /To allow it everywhere, even if another rule blocks other bots:
User-agent: Google-Extended
Allow: /Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.
How to tell if it's really Google-Extended
Google doesn't publish IP ranges or a reverse DNS domain for Google-Extended, so a request can't be proven to come from it. Anyone can send its user agent, so treat the name alone as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.
Other agents from Google
- Googlebot: AI and search crawler
- GoogleOther: AI training crawler
- Google-InspectionTool: AI and search crawler
- Storebot-Google: AI and search crawler
- AdsBot-Google: other bot
- Gemini-Deep-Research: AI assistant
- Google-CloudVertexBot: AI training crawler
- Google-Gemini-CLI: autonomous agent
- Google-NotebookLM: AI assistant
- GoogleAgent-Mariner: autonomous agent
- GoogleAgent-URLContext: AI assistant
Every Google agent, and a robots.txt for all of them →
Does Google-Extended visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from Google-Extended and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.