Ghost Agent Lab Free AgentScore Start 30-day trial

Agent directory ยท By company

Google's AI agents and crawlers

Google runs 15 agents that visit websites: 3 AI assistants, 3 AI and search crawlers, 2 autonomous agents, 6 AI training crawlers, 1 other bot. Here's what each one does, and how to let them in or keep them out.

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

Indexes pages so a search engine or AI search tool can find, cite and link to them. Allow it, unless you deliberately want to stay out of that search engine. Blocking it removes you from its results and answers.

  • Googlebot: Google's main crawler for Google Search. What it indexes also feeds Google's AI features in Search, such as AI Overviews.
  • Google-InspectionTool: Used by Google Search Console's URL Inspection and the Rich Results Test, when someone tests a page.
  • Storebot-Google: Crawls product pages, carts and checkouts for Google Shopping.

Autonomous agents

Uses a website like a person would, clicking, typing and filling in forms to finish a task for someone. Make your journeys work for it. Many browser agents look like an ordinary browser, so you can't reliably block them by name; test whether they can finish your key tasks instead.

  • Google-Gemini-CLI: Google's Gemini CLI, a coding agent in the terminal, fetching pages for a developer.
  • GoogleAgent-Mariner: Google's Project Mariner, a browser agent that carries out tasks on websites for a user.

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

  • Google-Extended: Not a crawler: a robots.txt name Google reads to decide whether content Googlebot collects may be used to train and ground Gemini models. Blocking it doesn't affect Google Search.
  • GoogleOther: A general-purpose Google crawler used by Google's product and research teams, separate from Google Search.
  • Google-CloudVertexBot: Crawls a site when its owner asks Google Cloud to, for example to build an AI agent on Vertex AI from the site's content.
  • Google-Firebase
  • GoogleOther-Image
  • GoogleOther-Video

Other bots

A well-known bot that isn't an AI agent, such as an ads checker or a link preview. Usually allow it: these bots support products you already use.

  • AdsBot-Google: Checks the quality of pages that Google Ads link to.

Robots.txt for every Google agent

To keep Google's training crawlers out but still let its assistants, search and agents in:

User-agent: Google-Extended
User-agent: GoogleOther
User-agent: Google-CloudVertexBot
User-agent: Google-Firebase
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
Disallow: /

To block every Google agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: Googlebot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: Google-InspectionTool
User-agent: Storebot-Google
User-agent: AdsBot-Google
User-agent: Gemini-Deep-Research
User-agent: Google-CloudVertexBot
User-agent: Google-Gemini-CLI
User-agent: Google-NotebookLM
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: Google-Firebase
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Google

Google publishes ways to check that a request really came from it:

Requests that use Google's names but fail these checks are impostors. Ghost Agent Lab checks every request this way. How to verify AI crawlers.

Which Google agents visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from them: what they read, what your site sent back, and whether each one was genuine. See agent traffic, or check whether AI agents can use your site for free.