Amazon's AI agents and crawlers

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

• Amzn-User: Fetches pages to answer questions people ask Amazon's assistants, such as Alexa.

AI and search crawlers

Indexes pages so a search engine or AI search tool can find, cite and link to them. Allow it, unless you deliberately want to stay out of that search engine. Blocking it removes you from its results and answers.

• Amazonbot: Amazon's crawler, used to improve Amazon's products and services, including answers from Alexa.
• amazon-kendra

Autonomous agents

Uses a website like a person would, clicking, typing and filling in forms to finish a task for someone. Make your journeys work for it. Many browser agents look like an ordinary browser, so you can't reliably block them by name; test whether they can finish your key tasks instead.

• AmazonBuyForMe: Amazon's Buy for Me agent, which can buy products from brands' own websites for Amazon shoppers.
• NovaAct: Amazon Nova Act, which developers use to build agents that carry out tasks in a web browser.

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

• bedrockbot

Robots.txt for every Amazon agent

To keep Amazon's training crawlers out but still let its assistants, search and agents in:

User-agent: bedrockbot
Disallow: /

To block every Amazon agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: Amazonbot
User-agent: AmazonBuyForMe
User-agent: Amzn-User
User-agent: NovaAct
User-agent: amazon-kendra
User-agent: bedrockbot
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Amazon

Amazon publishes ways to check that a request really came from it:

• Reverse DNS: names ending in crawl.amazonbot.amazon

Requests that use Amazon's names but fail these checks are impostors. Ghost Agent Lab checks every request this way. How to verify AI crawlers.