Agent directory · AI and search crawler
Storebot-Google
Crawls product pages, carts and checkouts for Google Shopping.
- Run by
- Type
- AI and search crawler
- Robots.txt name
Storebot-Google- Verifiable
- Yes
What Storebot-Google does
Crawls product pages, carts and checkouts for Google Shopping.
As an AI and search crawler, it indexes pages so a search engine or AI search tool can find, cite and link to them. The kinds of AI agent that visit websites.
Should you allow it?
Allow it, unless you deliberately want to stay out of that search engine. Blocking it removes you from its results and answers. A decision guide for blocking or allowing AI crawlers.
How to allow or block Storebot-Google
Storebot-Google reads the Storebot-Google group in your robots.txt. To block it everywhere:
User-agent: Storebot-Google
Disallow: /To allow it everywhere, even if another rule blocks other bots:
User-agent: Storebot-Google
Allow: /Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.
How it identifies itself
A typical user agent:
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36How to tell if it's really Storebot-Google
You can check a request that claims to be Storebot-Google against what Google publishes:
- IP ranges: developers.google.com/static/search/apis/ipranges/googlebot.json
- IP ranges: developers.google.com/static/search/apis/ipranges/special-crawlers.json
- IP ranges: developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json
- IP ranges: developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json
- Reverse DNS: the address resolves to a name ending in
googlebot.com,google.com,googleusercontent.com, and that name resolves back to the same address
Requests that use the name but fail these checks are impostors, usually scrapers trying to get past bot protection. Ghost Agent Lab checks every request this way and gives you rules to block impostors. How to verify AI crawlers.
Other agents from Google
- Googlebot: AI and search crawler
- Google-Extended: AI training crawler
- GoogleOther: AI training crawler
- Google-InspectionTool: AI and search crawler
- AdsBot-Google: other bot
- Gemini-Deep-Research: AI assistant
- Google-CloudVertexBot: AI training crawler
- Google-Gemini-CLI: autonomous agent
- Google-NotebookLM: AI assistant
- GoogleAgent-Mariner: autonomous agent
- GoogleAgent-URLContext: AI assistant
Every Google agent, and a robots.txt for all of them →
Does Storebot-Google visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from Storebot-Google and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.