Agent directory · AI training crawler
Google-CloudVertexBot
Crawls a site when its owner asks Google Cloud to, for example to build an AI agent on Vertex AI from the site's content.
- Run by
- Type
- AI training crawler
- Robots.txt name
Google-CloudVertexBot- Verifiable
- Not yet
What Google-CloudVertexBot does
Crawls a site when its owner asks Google Cloud to, for example to build an AI agent on Vertex AI from the site's content.
As an AI training crawler, it collects public pages that may be used to train AI models. The kinds of AI agent that visit websites.
Should you allow it?
A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else. A decision guide for blocking or allowing AI crawlers.
How to allow or block Google-CloudVertexBot
Google-CloudVertexBot reads the Google-CloudVertexBot group in your robots.txt. To block it everywhere:
User-agent: Google-CloudVertexBot
Disallow: /To allow it everywhere, even if another rule blocks other bots:
User-agent: Google-CloudVertexBot
Allow: /Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.
How to tell if it's really Google-CloudVertexBot
Our registry doesn't verify Google-CloudVertexBot against published IP ranges yet, so treat a request that uses the name as a claim, not proof: anyone can send its user agent. Check Google's documentation for how to verify it.
Other agents from Google
- Googlebot: AI and search crawler
- Google-Extended: AI training crawler
- GoogleOther: AI training crawler
- Google-InspectionTool: AI and search crawler
- Storebot-Google: AI and search crawler
- AdsBot-Google: other bot
- Gemini-Deep-Research: AI assistant
- Google-Gemini-CLI: autonomous agent
- Google-NotebookLM: AI assistant
- GoogleAgent-Mariner: autonomous agent
- GoogleAgent-URLContext: AI assistant
Every Google agent, and a robots.txt for all of them →
Does Google-CloudVertexBot visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from Google-CloudVertexBot and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.