The agents

AI agent. Software that uses an AI model to do something on the web for a person or a company: read a page, answer a question, compare products or complete a task. In our guides, “agent” covers everything from crawlers to agents that shop.

Training crawler. A bot that collects pages to train AI models, such as GPTBot or ClaudeBot. Blocking it doesn't stop AI assistants visiting for a person.

AI search crawler. A bot that indexes pages so AI search and assistants can cite and link to them, such as OAI-SearchBot or PerplexityBot.

Assistant fetcher. An AI assistant fetching a page in real time because a person asked it something, such as ChatGPT-User or Claude-User. Each visit stands in for a customer.

Browser agent. An agent that opens your site in a real browser and clicks, types and scrolls like a person, to book, sign up or buy. Also called a computer-use agent or autonomous agent.

User agent. The name a visitor sends with every request to say what it is, such as a browser version or GPTBot. Easy to fake, so it shouldn't be trusted on its own.

For how these differ and how to treat each, see the AI agents that visit your website.

Getting in: access

robots.txt. A file at the root of your site that tells crawlers and agents which pages they may visit. Well-behaved agents follow it; scrapers ignore it. See robots.txt for AI agents.

Bot protection. Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors. Set too strictly, they turn away AI assistants as well as scrapers. See bot protection and CAPTCHAs.

CDN. Content delivery network: the service that sits in front of your site and delivers it quickly around the world. It's often where bot rules live, and where agent traffic is recorded. See setting up your CDN for AI agents.

CAPTCHA or challenge. A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page. Agents can't pass them, so one on arrival or at checkout ends the visit.

Rate limit. A cap on how many requests a visitor can make in a given time. Fair limits stop scrapers without turning away agents. See rate limits for AI agents.

Verification. Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature. See how to tell if an AI crawler is real.

Impostor (spoofed agent). A request that uses a trusted agent's name but fails verification. Usually a scraper trying to get past bot protection.

Forward-confirmed reverse DNS. A verification check that an IP address's hostname belongs to the operator, and that the hostname points back to the same address.

Web Bot Auth. A new standard in which agents sign their requests cryptographically, so sites can prove who sent them. Our own scanner and Ghost Agents sign theirs. See why we sign every request.

llms.txt. A short Markdown file at /llms.txt that tells AI what your business does and links to your key pages. An emerging standard. See how to write an llms.txt file.

Guest checkout. Letting shoppers buy without creating an account. An agent buying for someone can't easily sign up for them. See guest checkout.

Understanding pages: readability

Raw HTML. The page exactly as your server sends it, before any JavaScript runs. It's all that many crawlers and assistant fetchers ever see. To look at it, use your browser's “View page source”, not the inspector.

Server-side rendering (SSR) and static generation. Ways of building pages so the content is already in the raw HTML, rather than added afterwards by JavaScript. See why AI agents can't see JavaScript-only content.

Structured data (schema.org, JSON-LD). Machine-readable facts in your page code, such as a product's name, price and stock, written in the shared schema.org vocabulary, usually as JSON-LD. Agents can read it without guessing. See product data AI shopping agents can read.

XML sitemap. A file listing the pages you want found, so crawlers and agents don't depend on following links. See XML sitemaps for AI agents.

Canonical URL. The one address you name as the main version of a page that can be reached in several ways. It stops agents treating duplicates as different products. See canonical URLs.

Alt text. A short text description of an image. Agents can't see pictures, so this is how they learn what a product photo shows. See alt text for AI agents.

Finding the way: navigability

Accessibility tree. The structured view of a page that screen readers and many browser agents use: its headings, links, buttons and form fields, with their names. If something isn't in it, many agents can't use it. See what an AI agent sees.

Accessible name. The name a button, link or field has in the accessibility tree, from its text, label or aria-label. An icon-only button with no name is invisible to agents.

Landmarks. Labels for the main regions of a page, such as header, navigation, main content and footer, that let agents skip straight to what they need.

Overlay. Anything that covers the page, such as a cookie banner, newsletter pop-up or chat window. If an agent can't close it, the journey ends there. See cookie banners and pop-ups.

WCAG. The Web Content Accessibility Guidelines, the standard for making sites usable by people with disabilities. Much of it helps agents too. See accessibility work is agent readiness work.

Getting it done: tasks and commerce

Journey. A task a visitor sets out to complete on your site, such as finding a product and reaching checkout, or booking a call. The journeys that make you money are the ones to test first.

Task completion. Whether an agent can actually finish a journey. It's the AgentScore category worth the most points, and the only way to measure it is to try. See how to test your key journeys.

MCP (Model Context Protocol). A standard way to give AI agents direct tools for your site, such as search, product details and cart, instead of making them click through pages.

WebMCP. A proposal for offering those same tools from inside your web pages, so a browser agent can call them while it's on your site. See MCP and WebMCP.

Agentic commerce. Purchases that an AI agent makes for a shopper. Protocols such as the Agentic Commerce Protocol, the Universal Commerce Protocol and AP2 aim to make that safe for buyer, store and payment provider. See agentic commerce protocols explained.

Measuring it

Agent readiness. How well AI agents can get into your site, understand it, find their way around it and complete tasks on it. See what is agent readiness?

AgentScore. Our free score out of 100 for agent readiness, across Access (25 points), Readability (25), Navigability (20) and Task completion (30). The free scan marks Task completion “not tested” and scores the other three. See how AgentScore works.

Check. One thing AgentScore tests, such as “Buttons and links have names”, reported as a pass, warning or fail, with who usually fixes it.

Ghost Agent. A real AI agent we send through a journey on your site, on a schedule, to see whether agents can complete it. You get a step-by-step replay of each run, and an alert when a journey breaks. See Ghost Agent.

Run and pass rate. A run is one attempt by one agent at a journey. AI agents don't behave the same way every time, so journeys are tried several times, and the pass rate is the share of runs that succeed.

Flaky. A journey where some runs pass and some fail. Agents can do it, but not reliably, which usually points to a slow page, a pop-up that appears sometimes, or an unclear control.

Agent traffic. Visits from AI agents of every kind. Most never run your analytics tag, so it's measured from server or CDN logs. See how to measure agent traffic.

AI referral. A person who arrives at your site by clicking a link in an AI assistant's answer. Unlike agent visits, these do show up in analytics. See tracking visits and sales from AI assistants.

Crawls per AI referral. How many pages an AI company's agents fetched for every visitor its assistant sent you. A low number means you get traffic back for what they read.

To see where your own site stands on each of these, run AgentScore. It's free and takes under a minute.

Why the difference matters

Each kind of agent does a different job for a different company, and brings you a different kind of value. A training crawler takes your content and may never send anyone back. An assistant fetching your returns policy is answering a customer's question right now. A browser agent on your checkout page is trying to place an order.

Treat them all as “bots” and you get one of two bad outcomes: block everything and disappear from AI answers and purchases, or allow everything without knowing what you've agreed to. The rest of this guide gives you the vocabulary to make a separate decision for each.

The five kinds at a glance

1. Training crawlers

Training crawlers collect large numbers of pages to train AI models. Examples include GPTBot, ClaudeBot and CCBot. They don't act for a particular person, and a visit from one doesn't mean anyone is looking for you.

• What they need: nothing special. They read your HTML like a search crawler.
• How to control them: robots.txt. They use their own names, so you can block them without blocking AI search or assistants. See robots.txt for AI agents.
• The decision: whether your content may be used for training is a business and legal choice, not a readiness one. AgentScore reports blocked training crawlers but doesn't take points off for them.

2. AI search crawlers

AI search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, build the index that AI search and assistants draw on when they answer questions and cite sources. They work much like Googlebot does for traditional search.

• What they need: your key content in the HTML your server sends, clear titles and headings, structured data, and a sitemap. Many don't run JavaScript. See why AI agents can't see JavaScript-only content.
• How to control them: robots.txt, and your CDN or bot protection, which has to let them through.
• Why allow them: if they can't read you, AI answers about your category are written from your competitors' pages.

3. Assistant fetchers

When someone asks ChatGPT, Claude or Perplexity “what's the return window at this store?” or “compare these two products”, the assistant may fetch the pages it needs there and then. Those requests use names like ChatGPT-User, Claude-User and Perplexity-User.

• What they need: fast, readable pages with the facts in plain text: prices, stock, shipping and returns. See shipping, returns and FAQ pages AI assistants can quote.
• How to control them: some operators treat these fetches like a person clicking a link and say robots.txt may not apply. Stopping them completely means a rule at your CDN or server.
• Why allow them: every one of these visits stands in for a customer asking about you. These are usually the visits you want most.

4. Browser agents

Browser agents go further: they open your site in a real browser and use it the way a person would, to search, choose a size, fill in a form or reach checkout. Several AI assistants now offer an agent mode that does this on a person's behalf, and businesses are starting to use them too.

• What they need: buttons, links and form fields with clear names, pop-ups they can close, no CAPTCHA in the way, and a checkout that doesn't force an account. They find their way around much as a screen reader does. See buttons, links and forms agents can use and how to make checkout work for AI shopping agents.
• How to recognize them: often you can't from the name alone, because many use an ordinary browser user agent. Some operators now sign their requests with Web Bot Auth so sites can tell who they are.
• Why they matter: they're the agents that complete tasks, so they're where agent readiness turns into revenue, and where a broken button or a bot challenge costs you an order.

5. Tool-using agents

Instead of reading pages, some agents call tools a site offers them directly: “search products”, “check stock”, “add to cart”. These tools are offered through the Model Context Protocol (MCP), WebMCP in the browser, or agentic commerce protocols for checkout.

• What they need: a set of well-described, safe tools, starting with read-only ones such as search and product details.
• Why it's worth planning for: tools are faster and more reliable than clicking through pages, and few sites offer them yet. See MCP and WebMCP and agentic commerce protocols explained.
• Where to start: most businesses get more from fixing the first four kinds first. Tools build on readable pages and clean product data, rather than replacing them.

How to see which ones visit you

Your web analytics probably shows almost none of this. Most agents never run the JavaScript tag analytics depends on, and the ones that do are often filtered out as bots. To see them, look at your server or CDN logs, where every request is recorded with its user agent and IP address. Then check that the visitors claiming to be well-known agents really are: names are easy to fake.

1. Count by kind, not by bot. Group visits into the five kinds above, so you can see how many customer-driven fetches you get compared with crawls.
2. Verify the big names. Check claimed agents against the IP ranges or signatures their operators publish. See how to tell if an AI crawler is real.
3. Watch for errors. A rise in blocked or failed requests from one kind of agent usually means a rule changed. See finding the errors AI agents hit.

Ghost Agent Labs does this for you from your CDN or server logs. See how to measure agent traffic.

A sensible starting policy

For a fuller decision guide by business type, see should you block AI crawlers? To see how your site treats each kind today, run AgentScore: its robots.txt check reads your rules as 21 AI agents would.

If you're still working out why agents get caught in the first place, start with Bot protection and CAPTCHAs: stop blocking the agents you want. This article is the practical follow-on: how to set the CDN up, and how to check it stays that way.

Start with a decision, not a setting

Before anyone changes a rule, agree which kinds of agent you want. Most businesses land somewhere like this:

Write the decision down. It's what your CDN admin implements, and it's what you check against later. Our guides to robots.txt for AI agents and whether to block or allow AI crawlers cover the trade-offs.

Use verified bot categories, not user agent strings

Most large CDNs and bot management products keep a list of verified bots: crawlers and agents whose requests they have confirmed really come from the operator they name. Many also group those bots into categories, so you can write one rule for "AI assistants" instead of maintaining a list of names yourself.

Build your rules on that verification, not on the User-Agent header. Anyone can send a request that says it's ChatGPT-User. A rule that skips bot checks for that string is an open door for scrapers. Our guide to telling real AI crawlers from fakes explains the methods CDNs use behind the scenes: published IP ranges, reverse DNS and signatures.

Turn on signature verification where you can

IP-based verification struggles with agents that run in the cloud, because their addresses change and are often shared. Web Bot Auth fixes that by having the agent sign each request with a private key, and publish the matching public key on its own domain. It builds on HTTP Message Signatures (RFC 9421), and is still being standardized, so support varies.

If your CDN can verify these signatures, turn it on and treat a valid signature as verification. If it can't yet, nothing breaks: signed requests are ordinary requests with extra headers. Our post Why we sign every request our agents send explains how it works in practice.

Get the rule order right

CDNs apply rules in a sequence, and the first matching action often wins. Most agent problems we see come from order, not from a missing rule. A sensible order looks like this:

1. Security rules for everyone. Managed WAF rules that catch attacks such as SQL injection should apply to every request, verified bots included. Verification proves who sent a request, not that the request is safe.
2. Hard blocks you need for legal or business reasons. For example, blocked training crawlers, or country restrictions you're required to enforce. See Geo-blocking, VPN blocks and AI agents for the country case.
3. Allow verified agents you want, by category or verified identity. "Allow" here should mean "skip bot challenges and bot scoring", not "skip all security".
4. Rate limits for everyone, including verified agents, set high enough for normal use. See rate limits for AI agents.
5. Bot scoring and challenges for whatever is left: unverified automation, impostors and suspicious traffic.

The common mistake is a broad "block AI bots" or "challenge automated traffic" rule placed above the verified-agent allow rule. The allow rule then never runs.

Challenge or block: choose deliberately

Most bot tools offer several actions. For a person they feel very different. For an AI agent, most of them end the same way.

So treat a challenge as a block when you're thinking about agents. Keep challenges for traffic you genuinely don't want, and for high-risk actions such as login, account creation and payment. Product, pricing, policy and help pages should open for verified agents without one.

Log enough to see what happened

When an assistant says it "couldn't access" your site, someone has to find out why. Make sure your CDN logs, or the samples it keeps, include for each request:

• The user agent, and whether the CDN treated the request as a verified bot (and in which category)
• The action taken (allowed, challenged, blocked, rate limited) and the ID of the rule that took it
• The response status code and the path
• The client country and network, if you use location rules

Before you switch a new rule to block, run it in log-only (sometimes called "simulate" or "count") mode for a few days and check what it would have caught. Then review blocked and challenged agent traffic regularly. Our guides to measuring AI agent traffic and finding the errors AI agents hit in your logs show what to look for.

Notes for common vendors

Products change quickly and menus move, so treat these as pointers, and check your vendor's current documentation for the exact names and the plan you're on.

• Cloudflare maintains a verified bots program and documents bot categories that you can reference in custom rules, including separate groupings for AI crawlers and other AI agents. It also offers settings aimed at blocking AI crawlers; check exactly which agents those cover before turning them on. Cloudflare has published work on Web Bot Auth, so check whether signature verification is available on your plan.
• Akamai bot management sorts known bots into categories and lets you set an action for each. Find where the AI-related categories sit and which action they currently get.
• Fastly offers bot management and WAF features with its own way of identifying known bots. Check how verified bots are flagged in its rules, and where that check runs relative to your other rules.
• AWS WAF has a bot control managed rule group that labels requests, including whether a bot is verified. Rules later in the web ACL can act on those labels, so order and labels together decide what agents get.

If you use a dedicated bot product in front of or behind your CDN, the same questions apply to it. Two layers mean two places an agent can be stopped.

Check it from the outside

A configuration that looks right in the console can still behave differently in practice. Test it as an agent would:

One caution: AgentScore's requests under those agent names aren't signed and don't come from those operators' networks, so a correctly configured CDN that only allows verified agents may still challenge them. That's the right behavior. Read a failure alongside your logs: if real, verified agent traffic is getting through, your setup is working. Our post on why our scanner wears other names explains the trade-off.

Where AI agents actually come from

When a person asks an assistant a question, the page request doesn't come from their phone or laptop. It comes from the assistant's servers. The same is true for AI search crawlers and for browser agents that run in the cloud. That has three consequences:

• The country is the server's, not the shopper's. Agent traffic is concentrated in the regions where AI companies run their infrastructure. Those are rarely the same as your customers' locations.
• The network is a data center. Agent requests come from cloud and hosting providers, the same networks that a lot of abusive automation uses. Reputation lists often score them as high risk.
• Addresses change. Agents that run on shared cloud infrastructure don't always come from fixed, published IP ranges.

None of this is a sign of bad intent. It's simply how AI agents work. But it means rules written with human visitors in mind can treat agents very differently from the people they're acting for.

Four rules that catch agents

Country blocks

Some sites block whole countries, often to cut fraud or because they don't ship there. If an assistant's servers sit in a blocked country, every request it makes for every shopper is turned away, including shoppers in countries you serve.

VPN, proxy and data center blocks

Blocking "anonymizing" networks or hosting providers is a common anti-fraud and anti-scraping setting. It catches nearly all AI agent traffic, because nearly all of it comes from hosting providers. It's one of the most common reasons an assistant can reach a competitor's site but not yours.

Automatic geo-redirects

Many international stores redirect visitors to a country site based on IP address. An agent fetching your UK product page for a UK shopper may be sent to your US site instead, then quote US prices, US stock and US delivery times. The answer looks confident and is wrong. Some redirects go to a country picker page, which leaves the agent with no product at all.

Location-based content

Some sites quietly change prices, currency, product ranges or policies based on the visitor's location, without changing the URL. An agent then describes a version of your store the shopper will never see.

What to do instead

The aim is to keep the protection you need while letting verified agents see the right content.

1. Separate legal requirements from preferences

Some location rules are required: sanctions, licensing, or content you're not allowed to show in certain places. Keep those, and get advice on how they apply to automated visitors. Many others are preferences, such as "we don't ship there, so block it". For those, ask whether blocking the server's location actually achieves anything. You already check the delivery address at checkout, which is where a shipping restriction belongs.

2. Let verified agents past location and network rules

Most CDNs and bot management tools can tell verified bots apart from other traffic. Add an exception so that verified AI assistants and AI search agents skip the country, VPN and data center rules that are about fraud or scraping. Put it in the right place in your rule order, and never base it on the user agent alone. Our guide to setting up your CDN for AI agents covers rule order, and telling real AI crawlers from fakes covers verification.

3. Let the URL decide the country, not the IP address

Give each country or language its own URL, such as northwind.example/en-gb/ or uk.northwind.example, and serve the content that URL promises, whoever asks. If you want to steer people to their local site, show a banner suggesting it instead of redirecting. Agents and people can then follow the link that matches the shopper. Mark the alternatives with hreflang so agents and search engines can find them:

<link rel="alternate" hreflang="en-gb" href="https://northwind.example/en-gb/boots/trail-runner/">
<link rel="alternate" hreflang="en-us" href="https://northwind.example/en-us/boots/trail-runner/">
<link rel="alternate" hreflang="x-default" href="https://northwind.example/boots/trail-runner/">

Our guide to languages, currencies and regions goes further on international setups.

4. Make the default page useful

Some agents will still land on your default site. Make sure it's a real store, not a country picker. State which country and currency the prices are in, and link clearly to the other country sites. Put the same information in your structured data, so an agent knows the price it read is in US dollars and not pounds. See machine-readable prices.

5. Use rate limits for the rest

For traffic you can't verify, from data centers or anywhere else, rate limits are usually a better tool than outright blocks. They stop bulk scraping while still letting through a small number of requests made on someone's behalf. See rate limits for AI agents.

How to find out if this affects you

An AgentScore scan helps too. Its requests come from cloud servers, like real agents' requests do, and it compares what AI agents get from your home page, product, pricing and cart pages with what a normal browser gets from the same servers. It can't show what visitors in every country see, so a block that only applies to certain countries may not show up. Pair it with a look at your rules and logs. Our guide to finding the errors AI agents hit in your logs shows how to pull out agent requests and their status codes.

A short checklist

What agents can and can't reach

Different kinds of agent meet a login wall in different ways:

So the question for each page is simple: if an agent can only see the public version, can it still answer the customer's question and move them forward?

What to keep public

These are the things agents are most often asked about. If they sit behind a login, the agent either says it doesn't know or quotes a competitor who shows them openly.

• Prices. "Sign in to see price" and "add to cart to see price" hide the one fact shoppers ask about most. If you can't show an exact price, show a starting price or a range. See machine-readable prices.
• Product information. Specifications, sizes, materials, compatibility and stock. A catalog that only appears after sign-in is invisible to agents.
• Policies. Shipping, returns, warranty and payment options. These decide many purchases, and agents quote them directly. See shipping, returns and FAQ pages AI assistants can quote.
• Plans and pricing for software. What each plan includes and costs, and whether there's a free trial. See agent readiness for software companies.
• Help content. Setup guides, FAQs and documentation. If your help center needs a login, assistants can't answer support questions about your product.
• Contact and location details. Opening hours, addresses and how to reach you.

Trade and wholesale businesses often hide everything, because prices vary by customer. Consider showing the catalog and list prices publicly, with a note that account holders get their own prices after signing in. The agent can then recommend you and send the buyer to log in for the final number.

What can stay behind the wall

Plenty of content belongs there: order history, saved addresses, account settings, customer-specific prices, members-only content and paid articles. The goal isn't to remove logins. It's to make sure the wall sits in front of private and paid things, not in front of the information a customer needs to decide.

Handing off to the person

When an agent does reach something that needs the customer, make the handoff smooth:

• Say what's needed in plain text. "Sign in to see your order history" is something an agent can relay. A login form with no explanation isn't.
• Return to the same place. After sign-in, send the person back to the page or cart they were on, with the cart intact, so the agent's work isn't lost.
• Don't require an account to buy. Guest checkout lets an agent finish a purchase without creating an account for someone. AgentScore checks for it in Shoppers can check out without an account, and our guide to guest checkout explains why it matters.
• Use proper status codes. Redirect to a login page, or return 401 or 403, rather than a 200 page that looks like content but isn't. Agents and crawlers can then tell "this needs a login" from "this page says nothing".
• Make sign-in forms agent-friendly. Labelled fields and standard autocomplete values help browser agents and password managers alike. See sign-up, booking and lead forms AI agents can finish.

Paywalls and structured data for publishers

Publishers face a specific version of this. You may want search engines to see the full article so they can rank it, while readers see a paywall. Showing crawlers different content from readers can look like cloaking. Google's documented answer is paywalled content structured data: you mark the article as not free, and point to the part that's behind the paywall.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "NewsArticle",
  "headline": "How trail running shoes are tested",
  "isAccessibleForFree": false,
  "hasPart": {
    "@type": "WebPageElement",
    "isAccessibleForFree": false,
    "cssSelector": ".paywalled-content"
  }
}
</script>

The cssSelector must match a class on the element that wraps the paid content. Google explains the details in its paywalled content guidelines, and the properties are defined at schema.org.

Two things to keep in mind:

• The markup describes access; it doesn't grant it. Whether you show full articles to any crawler, including AI search and training crawlers, is a business and licensing decision. Make it deliberately, per agent, and reflect it in robots.txt and your CDN rules.
• Give agents something to cite. A clear headline, a summary paragraph and the key facts above the paywall let assistants describe and link to your article even when they can't read it all. A paywall that hides everything, headline included, gets you nothing.

How to check your own site

Why business details matter to AI agents

People ask AI assistants questions like "is this a real shop?", "do they have a phone number?", "are they open on Sunday?" and "can I return it if it doesn't fit?". To answer, an agent has to find those facts somewhere on your site. Usually they're scattered: the phone number in the footer, the address on a contact page, the returns window in a policy page, the opening hours in an image.

An agent can often work it out, but every guess is a chance to get it wrong or to give up and recommend someone whose details are easier to confirm. Structured data about your business gives it one trusted summary, on the page it's most likely to visit first: your home page.

Product markup describes what you sell. Organization markup describes who is selling it. Agents use both. For products, see product data that AI shopping agents can read.

Pick the right type

Schema.org has a family of types for businesses. Use the most specific one that's true.

If you have an online store and physical shops, describe the company as an Organization or OnlineStore on the home page, and give each shop its own LocalBusiness (or Store) markup on its own location page, with its own address and hours.

A complete example

Here's JSON-LD for the home page of a fictional online coffee store. It goes in the page's HTML, inside the <head> or <body>:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "OnlineStore",
  "@id": "https://northwind.example/#organization",
  "name": "Northwind Coffee",
  "legalName": "Northwind Coffee Ltd",
  "url": "https://northwind.example/",
  "logo": "https://northwind.example/images/logo.png",
  "description": "Specialty coffee roasted to order and shipped across the US.",
  "email": "help@northwind.example",
  "telephone": "+1-555-010-0199",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "120 Harbor Street",
    "addressLocality": "Portland",
    "addressRegion": "OR",
    "postalCode": "97201",
    "addressCountry": "US"
  },
  "contactPoint": [{
    "@type": "ContactPoint",
    "contactType": "customer service",
    "email": "help@northwind.example",
    "telephone": "+1-555-010-0199",
    "availableLanguage": ["en"],
    "hoursAvailable": {
      "@type": "OpeningHoursSpecification",
      "dayOfWeek": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
      "opens": "09:00",
      "closes": "17:00"
    }
  }],
  "sameAs": [
    "https://www.instagram.com/northwindcoffee.example",
    "https://www.linkedin.com/company/northwind-coffee-example"
  ],
  "hasMerchantReturnPolicy": {
    "@type": "MerchantReturnPolicy",
    "applicableCountry": "US",
    "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
    "merchantReturnDays": 30,
    "returnMethod": "https://schema.org/ReturnByMail",
    "returnFees": "https://schema.org/FreeReturn",
    "merchantReturnLink": "https://northwind.example/returns"
  }
}
</script>

What each part tells an agent

• Name, legal name, URL and logo. Confirms which business this is, and that this website belongs to it. Use the name customers know you by in name, and your registered company name in legalName.
• Contact points. A ContactPoint for each way to reach you, with contactType saying what it's for ("customer service", "sales", "technical support"). Add the hours it's staffed and the languages it covers, so an agent can tell someone whether they'll get an answer today.
• Address. A full PostalAddress answers "where are they based?" and helps an agent judge shipping times and which country's consumer rules apply. Only include it if you're happy for it to be quoted.
• sameAs. Links to your official profiles elsewhere: social accounts, marketplace storefronts, a Wikipedia or business-register entry. They help an agent connect your site to what others say about you, and make it harder to confuse you with a similarly named business.
• Return policy. hasMerchantReturnPolicy on the organization sets a default for everything you sell. Google added support for return policies at the organization level, so you don't have to repeat it on every product. Product-level policies can still override it where they differ.

Opening hours for places customers visit

For a shop, showroom or other location, the most common question is "are they open now?". Use openingHoursSpecification on the LocalBusiness, one entry per set of days with the same hours:

"openingHoursSpecification": [
  { "@type": "OpeningHoursSpecification",
    "dayOfWeek": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
    "opens": "08:00", "closes": "18:00" },
  { "@type": "OpeningHoursSpecification",
    "dayOfWeek": "Saturday", "opens": "09:00", "closes": "16:00" }
]

Leave out days you're closed. For holidays, add an entry with validFrom and validThrough dates, so an agent doesn't send someone to a locked door on December 25. Add geo coordinates and a telephone for each location too.

Rules that keep it trustworthy

• Match what people see. The phone number, address, hours and return window in your markup must match your contact and policy pages. If they disagree, an agent can't tell which is right, and may say so.
• One source of truth. Generate the markup from the same settings as your footer and contact page, so a change in one place updates both. Hand-written markup goes stale the first time your hours change.
• Put it in the HTML. Markup added by a tag manager or other JavaScript after the page loads is invisible to agents that don't run JavaScript.
• Describe it once. Use a stable @id (like https://northwind.example/#organization) so other markup on your site, such as a product's brand or a WebSite's publisher, can point to the same organization instead of repeating it.
• Keep the policy pages too. Markup summarizes; your returns and shipping pages explain. Agents read both. See policy pages AI assistants can quote.

What AgentScore checks

The AgentScore check "Structured data describes your business and products" reads your home page's HTML for schema.org JSON-LD:

• It passes when JSON-LD is in the HTML before any JavaScript runs, and lists the types it found, such as Organization and WebSite.
• It warns if the JSON-LD only appears after JavaScript runs, or if the page only has microdata.
• It fails if there's no schema.org structured data on the home page at all.

The check confirms the markup is there and readable. It doesn't judge whether your hours or phone number are right, so validate the content yourself.

How to add it

1. See what you have. View your home page's source and search for application/ld+json. Many platforms and themes already output a basic Organization block with just a name and logo.
2. Collect the facts. Ask customer service for the contact details and hours they actually want shared, and check the return window against your current policy.
3. Extend the markup. Your developer can extend the theme template, or an SEO app or plugin can add the fields. Prefer one that reads from your store settings.
4. Validate. Run the page through the Schema Markup Validator, and Google's Rich Results Test for the fields Google uses.
5. Re-check after changes. Add "update the structured data" to the checklist for changing hours, phone numbers or policies.

Business details are one part of being readable to AI agents. For the rest, see the agent readiness guide, and for the details shoppers ask about most, reviews and ratings AI agents can read.

How AI agents compare products

A person browsing can look at a photo, skim the copy and get a feel for a product. An AI agent works differently. It turns the shopper's request into requirements (waterproof, under 500 grams, packable, under $200) and then checks each product page for evidence that the product meets them.

If your page says "featherlight protection for every adventure", the agent can't match that to "under 500 grams". If a competitor's page says "Weight: 410 g (size M)", it can. The agent recommends the product it can back up, and yours drops out, even if it's the better jacket.

This doesn't mean brand voice has to go. It means the facts need to be on the page too, written so they can be read and compared on their own.

Write specs as text, not images

Agents read text. Specifications that live only in an image, a size-chart graphic or a downloadable PDF are often invisible to them. The same goes for details that only appear when someone clicks a tab or accordion, if that content is loaded by JavaScript rather than present in the HTML. (Content hidden by CSS until clicked is fine; it's still in the page.)

Put the key facts in the page's HTML as plain text. A simple table works well, because each row is a clear pair of attribute and value:

<table>
  <caption>Specifications</caption>
  <tr><th scope="row">Weight</th><td>410 g (size M)</td></tr>
  <tr><th scope="row">Waterproof rating</th><td>20,000 mm hydrostatic head</td></tr>
  <tr><th scope="row">Shell fabric</th><td>100% recycled nylon, 3-layer</td></tr>
  <tr><th scope="row">Packed size</th><td>18 x 12 x 7 cm, packs into chest pocket</td></tr>
  <tr><th scope="row">Fit</th><td>Regular, room for a fleece underneath</td></tr>
</table>

A definition list (<dl>) works just as well. What matters is that each fact has a label next to it, in text.

Make attributes comparable

Comparison only works when the same attribute is described the same way across your catalog. Agree on a set of attributes for each category and fill them for every product.

Use the same attribute names, the same order and the same units on every product in a category. An agent comparing your three kettles should find "Capacity" in the same place on each page.

Units, materials and measurements

• Always give the unit. "Width: 60" means nothing. "Width: 60 cm" does. Where your customers use both systems, give both: "60 cm (23.6 in)".
• Say what was measured. "Weight: 410 g (size M)" or "Battery life: up to 14 hours (video playback, 50% brightness)". Conditions turn a claim into a fact.
• Name the materials. Give the fiber, metal or wood, the percentage in blends, and any certification by its proper name. "Vegan leather" is a category; "polyurethane-coated cotton" is a material.
• Give exact sizes. Link to a size guide in HTML text, not only an image, with body measurements per size.
• State what's in the box. Agents get asked "does it come with a charger?". Answer it in the description.

Compatibility and fit

For parts, accessories, consumables and refills, compatibility is the whole decision. A shopper asks "will this filter fit my Model X200?" and the agent needs a clear yes or no.

• List every compatible model by its full name and model number, not "fits most models".
• List known incompatible models where confusion is likely, such as the Pro version of a device.
• Put the list on the product page in text, not only in a separate compatibility tool that needs a form to be filled in.

In structured data, schema.org has properties such as isAccessoryOrSparePartFor and isConsumableFor that point from your product to the product it's used with.

Add the facts to structured data

Product JSON-LD is where agents look first for price and stock. It can carry descriptive attributes too, so they don't have to be read out of your layout:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Ridgeline Packable Rain Jacket, Men's",
  "sku": "NW-RJ-M-BLU",
  "material": "100% recycled nylon",
  "color": "Blue",
  "size": "M",
  "weight": { "@type": "QuantitativeValue", "value": 410, "unitCode": "GRM" },
  "additionalProperty": [
    { "@type": "PropertyValue", "name": "Waterproof rating",
      "value": 20000, "unitText": "mm" },
    { "@type": "PropertyValue", "name": "Packable", "value": true }
  ],
  "offers": {
    "@type": "Offer",
    "price": "189.00",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  }
}

Use additionalProperty for attributes schema.org has no dedicated property for. The values must match what the page says. For the full product markup, including variants, returns and shipping, see product data that AI shopping agents can read.

Avoid vague copy

Marketing copy still has a job: it persuades the person who reads it. The problem is when it replaces facts instead of sitting beside them. Watch for these:

• Superlatives without evidence. "Best-in-class", "industry-leading", "unbeatable". An agent can't verify them, so it ignores them.
• Relative claims. "30% lighter" than what? Give the absolute number.
• Copy shared across a range. If ten products share the same paragraph, an agent can't tell them apart. Lead with what makes each one different.
• Manufacturer text pasted unchanged. It's often thin, and the same on every store selling that product. Add your own facts.

A good pattern: one or two sentences of plain summary (what it is, who it's for, the one or two facts that matter most), then your brand copy, then the full specification table.

What AgentScore checks

AgentScore doesn't grade the wording of your descriptions. On a product page, it checks that schema.org Product data gives price, currency and availability ("Product pages give price and stock in a form agents can read"), and that prices are in the page HTML. To see how a real agent handles your descriptions, run a journey with Ghost Agent, such as "find a waterproof jacket under 500 grams", and read the step-by-step replay. See testing journeys with AI agents.

Where to start

1. Pick your best-selling category and list the five to ten attributes shoppers compare.
2. Audit ten product pages: is each attribute there, in text, with a unit?
3. Fill the gaps in your product information system or platform, not in each page by hand, so the data reaches every channel.
4. Add the specification table to the product page template and the same attributes to your Product JSON-LD.
5. Repeat for the next category.

Shoppers also lean on prices, variants and reviews when comparing; see machine-readable prices, variant pickers AI agents can use and reviews and ratings AI agents can read.

Why reviews matter to AI agents

AI agents use reviews in two ways. The rating is a quick signal when they rank options: a 4.7 from several hundred reviews reads differently from a 4.7 from three. The review text answers specific questions the product description doesn't: "does it run small?", "is it loud?", "how long did the battery last after a year?".

If an agent can't see your reviews, it has nothing to weigh. It may fall back on reviews it finds elsewhere, or simply prefer a competitor whose ratings it can confirm.

Put reviews in the HTML

Most stores use a review app or service. Many of these load reviews with JavaScript after the page opens, from a separate server. A person sees them a moment later. An AI agent that reads the HTML without running JavaScript sees an empty space where the reviews should be, and many agents work that way.

• Check the source. Open a product page, view the page source (not the browser's inspector, which shows the page after JavaScript), and search for a phrase from one of your reviews. If it isn't there, agents that skip JavaScript don't see it.
• Ask your provider about server-side output. Many review platforms offer a way to include the rating and the most recent or most helpful reviews in the page HTML, through a platform integration or an API. Turn it on if yours does.
• At minimum, render the summary. The average rating and review count, in text near the product name, plus a handful of reviews. "Rated 4.6 out of 5 from 318 reviews" is easy for any agent to read.
• Link paginated reviews. If reviews run to several pages, use ordinary links between them so agents can follow them.

For the wider problem of content that only appears after JavaScript runs, see why AI agents can't see JavaScript-only content.

Mark up ratings and reviews

Schema.org gives agents the same facts in a standard form. On a product page, add aggregateRating for the summary and review for individual reviews, inside your Product JSON-LD:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Ethiopia Yirgacheffe Whole Bean Coffee, 340 g",
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.6",
    "bestRating": "5",
    "reviewCount": "318"
  },
  "review": [{
    "@type": "Review",
    "author": { "@type": "Person", "name": "Dana R." },
    "datePublished": "2026-09-14",
    "reviewRating": { "@type": "Rating", "ratingValue": "5", "bestRating": "5" },
    "reviewBody": "Bright and floral as a pour-over. Arrived two days after roasting."
  }],
  "offers": {
    "@type": "Offer",
    "price": "18.00",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  }
}

Use reviewCount for reviews with text, or ratingCount if you count star-only ratings too. Say what scale you use with bestRating if it isn't 1 to 5.

The rules for review markup

Search engines publish rules for review markup, and following them is the safest guide for AI agents too. The key ones from Google's review snippet guidelines:

• Only mark up reviews people can see. Every review and rating in the markup should be readable on the same page. Markup for reviews hidden somewhere else, or that don't exist, breaks the rules and can get your structured data ignored.
• Mark up reviews of a specific thing. A product, a recipe, a course, a book. Not a category page summing up many products.
• Don't add review markup about your own business on your own site. Google calls reviews that a business hosts about itself, on its own site, "self-serving", and doesn't show review stars for them on LocalBusiness and Organization pages. The safe reading is: put ratings on your products, and leave your company-level reputation to independent review sites.
• Keep the numbers in step. The rating and count in the markup should match what the page shows, and update when new reviews come in.

Google's rules apply to its search results, not to every AI agent. But they describe what makes review data trustworthy, and agents that cross-check your markup against the page will notice the same problems.

Honesty is the strategy

AI agents read across many sources. If your site shows a perfect 5.0 and independent review sites say 3.2, the gap is itself a signal, and an agent may say so to the shopper. The only durable approach is reviews that reflect what customers actually think.

• Publish the negative reviews. Hiding everything below four stars makes the remaining rating less believable, to people and to agents. In the US, the FTC's rule on fake reviews and testimonials covers suppressing negative reviews as well as faking positive ones. Check the rules where you sell.
• Label incentives. If a reviewer got the product free or was rewarded for reviewing, say so next to the review.
• Mark verified buyers. A "verified purchase" label, applied honestly, helps agents weigh reviews.
• Reply in public. A clear reply to a complaint ("we've changed the packaging since March") is information an agent can pass on.
• Don't pool unrelated products. Showing reviews of an old model, or of the whole range, on a new product's page misleads. Pool reviews only across true variants, like sizes and colors of the same item.

What AgentScore checks

AgentScore doesn't have a dedicated reviews check. Its structured data checks look for schema.org JSON-LD on your home page, and for Product data with price, currency and availability on a product page, and flag markup that only appears after JavaScript runs. If your review app adds its markup with JavaScript, the same problem applies to your ratings. To see what an agent actually makes of your reviews, run a Ghost Agent journey such as "find the best-rated coffee under $20 and add it to the cart" and read the replay.

A short checklist

1. View the source of a product page. Are the rating, review count and some review text in the HTML?
2. If not, turn on your review provider's server-side or SEO output, or ask your developer to render a summary.
3. Check the Product JSON-LD includes aggregateRating that matches the page.
4. Remove any rating markup about your business as a whole from your own site.
5. Review your moderation policy: are you publishing negative reviews and labeling incentives?
6. Validate with Google's Rich Results Test.

Reviews work best alongside clear facts. See product descriptions AI agents can compare and product data that AI shopping agents can read.

Why duplicate URLs confuse AI agents

When an AI agent reads your site, or when an AI search service builds the index an assistant draws on, it meets your pages by URL. If the same jacket appears at five URLs, it may:

• treat them as five products, and compare your jacket with itself;
• quote a price or stock level from an old copy it read weeks ago;
• link a shopper to a version with a stale campaign tag, an odd filter or a variant they didn't ask for;
• spend its limited time on your site reading copies instead of finding new products.

The fix is not to make duplicates disappear, which is often impossible, but to say clearly which URL is the main one, and to make every other signal agree.

Where duplicates come from

Set a canonical URL on every page

A canonical link is one line in the page's <head> that names the preferred URL for the content:

<link rel="canonical" href="https://northwind.example/products/rain-jacket">

Put it on every indexable page, including on the canonical page itself (a self-referencing canonical). Rules that keep it reliable:

• Use the full, absolute URL. Include https:// and the host, not just the path.
• Put it in the HTML. A canonical tag added by JavaScript may never be seen by agents and crawlers that read the raw page.
• One per page. Two different canonical tags on a page is a common theme or app conflict. Search engines may ignore both.
• Point to a page that works. The canonical should return 200, not redirect, 404 or carry a noindex.
• Don't canonicalize everything to the home page. Each distinct page should point at itself, or at its true duplicate.

Search engines treat rel="canonical" as a strong hint rather than a command, and the same is likely true of AI services that build on them. That's why the other signals below need to agree with it.

Parameter URLs

Tracking parameters like utm_source and click IDs never change the content, so the page should always name the clean URL as canonical. Sort orders and session IDs are the same.

Filters are a judgment call. A filtered listing such as "blue jackets" may be worth keeping as its own page if people search for it. If so, give it a clean path (/jackets/blue), its own title and a self-referencing canonical. Most combinations of filters, though, should point back to the unfiltered category. For more on filters agents can use, see site search and filters for AI agents.

Product variants

Variants are where stores most often go wrong. There are two reasonable patterns:

• One page for the product. Every variant URL (?variant=4417, ?color=blue) has a canonical pointing to the main product URL. This suits products where variants differ only in size or color and share a description. Make sure the main page lists every variant's price and availability, ideally in ProductGroup markup, so the agent doesn't lose them.
• A page per variant. Each variant has its own URL with a self-referencing canonical, its own title ("Rain Jacket, Men's, Blue") and its own price. This suits variants that are really different products, with different specs or prices.

What doesn't work is a mix: variant URLs that canonicalize to the main page, while the main page only shows the default variant's price and stock. An agent is then told the blue jacket is the same page as the red one, but can't find the blue one's details. See size, color and variant pickers AI agents can use.

http, https and www: redirect, don't just canonicalize

For duplicates that are purely technical, a canonical tag isn't enough. Send a permanent (301 or 308) redirect so that every visitor, person or agent, lands on one version:

• http:// redirects to https://.
• Pick www.northwind.example or northwind.example, and redirect the other.
• Pick trailing slash or no trailing slash, and redirect the other.
• Redirect old product URLs to their replacements after a migration, not to the home page.

Redirect in a single hop. Chains (http to https, then to www, then to a trailing slash) slow every request down, and some clients stop following after a few hops. Avoid redirects that depend on JavaScript or a meta refresh; agents that don't run scripts get stuck on the first page. And make sure region or language redirects don't trap agents, as covered in international sites for AI agents.

Make every signal agree

Agents and crawlers look at more than the canonical tag. Each of these should use the same, canonical URL:

• Your XML sitemap. List only canonical URLs that return 200. No parameter URLs, no redirecting URLs, no http versions. See XML sitemaps for AI agents.
• Internal links. Navigation, product grids and breadcrumbs should link to the canonical URL directly, not to a version that redirects or carries parameters.
• Structured data. The url in your Product and Offer JSON-LD should match the canonical.
• og:url and feeds. Social tags and product feeds should use the same address.
• hreflang tags, if you have them, should point to canonical URLs of each language version.

What AgentScore checks

AgentScore doesn't have a dedicated canonical check. The "Valid sitemap" check confirms you have a sitemap it can read (found through robots.txt, /sitemap.xml or /sitemap_index.xml) and counts its entries; it doesn't judge whether each URL is canonical. AgentScore also uses your sitemap and home page links to find a product page to test, so a sitemap full of redirecting or parameter URLs makes that harder for any agent.

A quick audit

1. Open a product page with ?utm_source=test on the end. View the source and check the canonical points to the clean URL.
2. Do the same for a variant URL and a filtered category. Is the result the pattern you intended?
3. Type http:// and the non-preferred host into a browser. Do they each redirect, in one hop, to the right page?
4. Open your sitemap and spot-check ten URLs: do they all return 200, and match their own canonical?
5. Check your main navigation links for parameters or redirects.

Canonical URLs are housekeeping, but they decide which version of your page agents read and repeat. For the bigger picture, see the agent readiness guide.

What can go wrong

International sites are built for people, who can see a flag in the corner, read a price symbol and click "change country". AI agents work from the page's code and text, often from a server in another country, and usually without the browser settings that a person's visit carries. Common failures:

• The agent is redirected to the wrong country store based on where its server is, and can't get back.
• A price shows "$" with no sign of whether it's US, Canadian or Australian dollars.
• Three language versions of the same page look like three unrelated pages, or like duplicates.
• The agent tells a shopper a product is available, when you don't ship to their country.

Declare the language with the lang attribute

Every page should say what language it's in, on its <html> tag:

<html lang="en-GB">

Use a language code, optionally with a region: en, en-GB, fr-CA, de-CH. It tells agents, screen readers and translation tools how to read the page. Make sure it changes with the language version. A French page that still says lang="en", because the theme hard-codes it, is surprisingly common. The AgentScore check "Clear page title, description, and headings" flags a missing lang attribute on your home page; see titles, descriptions and headings.

Connect language and country versions with hreflang

hreflang links tell agents and search engines that several pages are versions of the same content for different languages or regions, and which one is for whom. Each version lists all the others, and itself:

<link rel="alternate" hreflang="en-us" href="https://northwind.example/us/products/rain-jacket">
<link rel="alternate" hreflang="en-gb" href="https://northwind.example/uk/products/rain-jacket">
<link rel="alternate" hreflang="fr-fr" href="https://northwind.example/fr/produits/veste-pluie">
<link rel="alternate" hreflang="x-default" href="https://northwind.example/products/rain-jacket">

• Make it two-way. If the US page lists the UK page, the UK page must list the US page. One-way links are usually ignored.
• Use canonical URLs. Each href should be the canonical URL of that version, returning 200, not a redirect. See canonical URLs and duplicate pages.
• Don't canonicalize across languages. The French page's canonical should point to the French page, not the English one.
• Add x-default. It names the page for everyone else, typically a global version or a country picker that works without redirects.
• Use correct codes. Language first, then optional region: en-gb, not uk or gb-en.

You can put hreflang in the page <head>, in HTTP headers, or in your XML sitemap. Pick one method and keep it consistent.

Region redirects that trap agents

Many stores redirect visitors automatically based on their IP address or browser language. For people, it saves a click. For AI agents it can be a dead end. Agents often run from cloud data centers, so the location you detect is the server's, not the shopper's. An agent helping someone in Berlin may be sent to your US store, and if every attempt to open /de/ bounces back to /us/, it can never read the German prices.

• Suggest, don't force. Show a banner ("Looks like you're in Germany. Go to the German store?") instead of redirecting.
• Never redirect away from an explicit country URL. If a request asks for /de/products/rain-jacket, serve it.
• Let the choice stick in the URL. Country and language in the path or subdomain work for agents. A choice stored only in a cookie doesn't, because many agents don't keep cookies between requests.
• Don't block whole regions without a reason. Blocking traffic from countries you don't sell to can also block the data centers AI agents run from. See geo-blocking, VPN blocks and AI agents.

Make the currency unambiguous

A symbol alone is often not enough. "$" is used by more than a dozen currencies, and "kr" by several. Show the currency code where there's any doubt (US$189 or 189.00 USD), and always put it in your structured data:

"offers": {
  "@type": "Offer",
  "price": "175.00",
  "priceCurrency": "GBP",
  "availability": "https://schema.org/InStock"
}

priceCurrency takes a three-letter ISO 4217 code: USD, GBP, EUR, CAD. Each country version should show its own price and currency in both the page and the markup. Say whether prices include sales tax or VAT, since that differs by country. More in machine-readable prices.

Say where you ship

"Do they deliver to Canada?" is a question an agent should be able to answer before sending someone to your checkout. Make it easy:

• A shipping page in plain text listing the countries you ship to, with costs and delivery times for each. Not only a dropdown at checkout.
• Product-level restrictions on the product page. If an item can't ship to some countries, say so on that page.
• Shipping details in structured data. Schema.org's shippingDetails on an Offer can describe where you deliver, at what cost, and how long it takes:

"shippingDetails": {
  "@type": "OfferShippingDetails",
  "shippingDestination": { "@type": "DefinedRegion", "addressCountry": ["GB", "IE"] },
  "shippingRate": { "@type": "MonetaryAmount", "value": "4.95", "currency": "GBP" },
  "deliveryTime": {
    "@type": "ShippingDeliveryTime",
    "handlingTime": { "@type": "QuantitativeValue", "minValue": 0, "maxValue": 1, "unitCode": "DAY" },
    "transitTime": { "@type": "QuantitativeValue", "minValue": 2, "maxValue": 4, "unitCode": "DAY" }
  }
}

Returns can differ by country too; give each MerchantReturnPolicy its applicableCountry. See Organization and LocalBusiness data and policy pages AI assistants can quote.

A checklist for each country version

1. The <html lang> attribute matches the page's language.
2. hreflang links connect every version both ways, with an x-default.
3. Each version's canonical points to itself.
4. Opening a country URL directly serves that country's page, with no forced redirect.
5. Prices are in the HTML, with an unambiguous currency, and priceCurrency matches.
6. The countries you ship to are listed in text, and in shippingDetails where you can.

If your country stores run on separate domains, such as northwind.example and northwind-uk.example, scan each one with AgentScore to compare them. To check a real journey in another market, such as "find a rain jacket and get to checkout on the UK store", run it with Ghost Agent and read the replay.

Why variant pickers trip agents up

A person sees a row of colored circles, notices a thicker border around one, and sees a crossed-out "XL". A browser agent mostly reads the page's structure: each control's role, its name and its state. If the swatch is a <div> with a background color, it has no role and no name. If "selected" is shown only with a border, there's no state to read. The agent may click the right circle by chance, click the wrong one, or not realize there's a choice to make at all.

The cost is direct. An agent that can't choose a size either gives up and tells the shopper it couldn't complete the task, or adds the wrong item to the cart. Both are lost or returned orders. And the same problems affect shoppers who use a keyboard or a screen reader, so fixing them helps people too.

Agents need four things from a picker: what each option is called, which one is chosen, which ones can't be bought, and confirmation that the page changed after they chose.

1. Make each option a real control with a name

The simplest, most reliable picker is a group of native radio buttons. Only one option can be chosen, the browser handles the keyboard, and every agent knows what a radio button is. You can style the label to look like a swatch or a size box.

<fieldset>
  <legend>Color</legend>
  <label class="swatch">
    <input type="radio" name="color" value="navy" checked>
    <span class="dot" style="background:#1f2a44" aria-hidden="true"></span>
    Navy
  </label>
  <label class="swatch">
    <input type="radio" name="color" value="olive">
    <span class="dot" style="background:#5b5e3a" aria-hidden="true"></span>
    Olive
  </label>
</fieldset>

• Name every option. The color name, such as "Navy", should be the control's name, as visible text or at least an aria-label. A hex code or an image file name isn't a name.
• Name the group. A <fieldset> with a <legend> ("Color", "Size") tells the agent what the choice is about.
• Use the words shoppers use. "Medium" or "M" is fine. An internal code like "SZ-03" isn't. If sizes need context ("UK 8 / US 4"), put it in the name.
• Keep swatch images as decoration. If a swatch is a photo of the fabric, give the control the color name and the image empty alt="", so the name isn't read twice.

If your design uses buttons rather than radios, that works too, as long as they are real <button> elements with names. AgentScore's checks that buttons and links have names, and that clickable things are real buttons and links, look for exactly these problems, though they run on your home page. A product page needs the same care.

2. Show which option is selected, in the code

Selection is the detail most often shown only visually. Radio buttons report it for free through checked. Custom buttons have to say it themselves:

<div role="group" aria-labelledby="size-label">
  <span id="size-label">Size</span>
  <button type="button" aria-pressed="false">S</button>
  <button type="button" aria-pressed="true">M</button>
  <button type="button" aria-pressed="false">L</button>
</div>

• Use aria-pressed="true" on the chosen button and "false" on the others, and update them when the choice changes.
• Alternatively, use role="radio" with aria-checked inside a role="radiogroup". It's more work to get the keyboard right, which is why native radios are usually the better choice.
• Repeat the current choice in text near the picker ("Color: Navy"). It gives agents and people a plain confirmation.

3. Mark out-of-stock options clearly

Hiding a sold-out size makes the shopper wonder whether it ever existed. Showing it with only a strike-through or faded color leaves an agent free to choose it, then fail at "Add to cart". Say it in words and in the control's state.

If you offer back-in-stock alerts, make that a separate, named button ("Email me when XL is back"), so an agent can tell it apart from buying.

4. Give each variant its own URL

Many AI assistants fetch pages rather than click through them. They can't press a swatch, but they can open a link. If each variant has its own address, an assistant can point a shopper straight to "the navy jacket in medium", and a browser agent can recover if a click doesn't take.

https://northwind.example/products/field-jacket?color=navy&size=m
https://northwind.example/products/field-jacket?variant=40318

• Update the address when a shopper picks an option, so the current view always has a link.
• Make the URL load with that variant already selected, its price and stock shown, when opened fresh.
• Point the canonical URL at the main product page unless variants are really separate products. See canonical URLs and duplicate pages.

5. Update price, stock and photos where agents can see it

Choosing a larger size or a different material often changes the price. If the new price only appears in an image, or changes quietly in a corner of the page, an agent may report the old one to the shopper.

• Show price and availability as text, and update them when the variant changes.
• Wrap the price and stock line in a live region (aria-live="polite"), so the change is announced to assistive tech and is easier for agents reading the page structure to notice.
• Keep "Add to cart" a real button with the same name throughout. If it's disabled until a size is chosen, say so next to it: "Choose a size". AgentScore's add-to-cart check accepts a button that's disabled until options are chosen, because agents can handle that, as long as they can tell what's missing.

6. Describe variants in your structured data

Agents that read your page HTML rather than click it rely on structured data for the full list of options. schema.org has a ProductGroup type for this: one group, with variesBy naming the dimensions and hasVariant listing each variant as a Product with its own offer. Google documents the pattern in its product variant structured data guide.

{
  "@context": "https://schema.org",
  "@type": "ProductGroup",
  "name": "Field jacket",
  "productGroupID": "FJ-100",
  "variesBy": ["https://schema.org/color", "https://schema.org/size"],
  "hasVariant": [{
    "@type": "Product",
    "sku": "FJ-100-NAV-M",
    "color": "Navy",
    "size": "M",
    "url": "https://northwind.example/products/field-jacket?variant=40318",
    "offers": {
      "@type": "Offer",
      "price": "189.00",
      "priceCurrency": "USD",
      "availability": "https://schema.org/InStock"
    }
  }]
}

AgentScore's "Product pages give price and stock in a form agents can read" check reads offers inside ProductGroup and hasVariant, as well as on a plain Product. For more on the fields that matter, see product data AI shopping agents can read and prices AI agents can read.

How to test your pickers

1. Use only the keyboard. Tab to the color and size pickers, choose an option, and add to cart. If you can't, many agents can't either.
2. Inspect the accessibility tree. In Chrome's developer tools, the Accessibility pane should show each option as a radio or button with a name like "Navy", and its checked or pressed state.
3. Pick a sold-out size. The page should tell you, in words, before you reach the cart.
4. Copy the URL after choosing. Open it in a private window. The same variant should be selected.
5. Run a journey. AgentScore's free scan doesn't choose options on your product pages, so the best test is an agent trying it. A Ghost Agent can run "find the field jacket, choose navy in medium, add it to the cart" on a schedule and replay each step when it gets stuck. See testing key journeys with AI agents.

For the rest of the page's controls, see buttons, links and forms agents can use. For what happens after "Add to cart", see carts AI agents can manage.

What agents see on a long list

Different agents meet a category page in different ways:

• Assistants that fetch pages, such as ChatGPT or Perplexity looking something up for a person, usually read the HTML your server sends. They don't scroll, and often don't run JavaScript. They see the first batch of products and nothing more.
• AI search crawlers also read HTML and follow links. If page two has no link, it doesn't exist for them.
• Browser agents can scroll and click, but each scroll is a step that takes time. They can't always tell whether more products are coming or the list has ended.

So a category of 300 products that shows 24 at a time, with more loading on scroll, can look like a category of 24. If the product a shopper asked for is number 61, the assistant may tell them you don't sell it.

Give every page of results its own URL

The foundation is the same for infinite scroll, "load more" and numbered pages: each chunk of results should exist at its own address, and that address should load those results when opened fresh.

https://northwind.example/collections/coffee
https://northwind.example/collections/coffee?page=2
https://northwind.example/collections/coffee?page=3

• Use query parameters or paths, not fragments. ?page=2 and /page/2 are separate URLs. #page=2 is not: servers never see what follows the #, so it can't return different results.
• Render the results in the HTML. ?page=2 should arrive with its products already in it, not an empty shell that fills in later. See why AI agents can't see your JavaScript-only content.
• Keep the order stable. Page two should show the same products whether you arrive from page one or open it directly. Random or personalized ordering makes pages unreliable to follow.
• Carry filters and sort along. ?roast=dark&sort=price-asc&page=2 should keep both. Our guide to site search and filters AI agents can use covers filter URLs in detail.

Google's guidance on pagination and incremental page loading takes the same approach, and it works for AI agents for the same reasons.

Link the pages together with real links

Unique URLs only help if agents can find them. Each page should link to the next one, and ideally to the previous one and a few page numbers, with ordinary <a href> links in the HTML.

<nav aria-label="Pagination">
  <a href="/collections/coffee?page=1">Previous page</a>
  <a href="/collections/coffee?page=1">1</a>
  <a href="/collections/coffee?page=2" aria-current="page">2</a>
  <a href="/collections/coffee?page=3">3</a>
  <a href="/collections/coffee?page=3">Next page</a>
</nav>

• Wrap the links in <nav> with a name like "Pagination", so agents can tell it apart from the main menu.
• Name arrow links in words ("Next page"), not just "›".
• Mark the current page with aria-current="page".
• Say how many products there are in text, such as "Showing 25–48 of 312". It tells an agent there's more to see.

Make "load more" a button, backed by a link

"Load more" is a good middle ground for shoppers, and it can work well for agents if it's built as two things at once: a real button for people and browser agents, and a real link for everyone else.

<a class="load-more" href="/collections/coffee?page=3">
  Load more products (264 remaining)
</a>

Start with a plain link to the next page. Then let JavaScript take over: intercept the click, fetch the next page, append the products, and update the address with history.pushState so the URL matches what's on screen. If the script doesn't run, or the agent doesn't run scripts, the link still goes to page three. If you prefer a <button>, keep the numbered page links alongside it so there's still a route.

• Give the button a specific name: "Load more products", not "More" or an arrow icon.
• Announce new results with a live region ("24 more products loaded") so the change is easier for agents and assistive tech to notice.
• Move keyboard focus to the first new product, or leave it on the button, but don't send it back to the top of the page.

Infinite scroll: keep it, but add pages underneath

If you want true infinite scroll, treat it as an enhancement over paginated pages. As each new batch loads, update the URL to the page number the shopper has reached. Keep the paginated links in the HTML, even if they're visually tucked away at the bottom. And make sure the footer is reachable: if the page grows every time someone nears the bottom, neither people nor agents can reach your contact, shipping and returns links.

What about rel="next" and rel="prev"?

For years, <link rel="next"> and rel="prev" in the page head were the standard way to tell search engines that pages belonged in a series. Google said in 2019 that it no longer uses them for indexing. Other search engines and tools may still read them, and they're part of HTML, so they do no harm. But don't rely on them: an agent following links needs the visible <a href> links in the page.

Two related head tags matter more:

• Canonical. Give each paginated page its own canonical URL. Pointing ?page=2 at page one tells crawlers to ignore the products on page two. See canonical URLs and duplicate pages.
• Robots meta. Don't mark paginated pages noindex by default. Pages that stay out of the index can, over time, get less attention from crawlers, and so can the products linked from them.

Back it up with a sitemap

Even with perfect pagination, some products sit deep in a long list. Your XML sitemap should list every product page directly, so crawlers don't depend on paging through categories to find them. AgentScore's "Valid sitemap" check confirms you have one. See XML sitemaps for AI agents. The sitemap is the safety net. The links are still the main route.

How to check your category pages

1. View the source of a category page and search for a product name from further down the list. Then look for a link to page two.
2. Open ?page=2 directly in a private window. It should show the second set of products, not the first and not an empty page.
3. Turn off JavaScript and try to reach the last page of a category.
4. Check the footer is reachable on a category page with infinite scroll.
5. Run AgentScore. It has no pagination check, but "Content loads without JavaScript" and "Key pages are linked from the home page" show whether HTML-only agents can see your content and reach your products.

For the bigger picture of how agents move around a site, see What is agent readiness?

Why errors matter more for agents

An agent working for someone wants to finish the task. When a form doesn't go through, it needs to answer three questions: did something go wrong, which field caused it, and what should it enter instead? People answer those from visual cues, such as a red border, a shake, an icon or a message that flashes up and disappears. Agents mostly read the page's text and structure, and may not catch any of those cues.

A clear error turns a failed attempt into a quick correction. An unclear one ends the journey, and you lose the order or sign-up without anyone telling you. The same fixes help people who use screen readers, and anyone filling in a form on a small screen.

Write errors that say what to do

Start with the words. An agent passes the message on, or acts on it, exactly as written.

• Name the field and the problem, and give an example of what's accepted.
• Say whether it's something the shopper can fix, or something on your side that may work later.
• Don't rely on color, icons or a border alone. If it isn't in text, assume an agent won't see it.
• Keep what was entered. Clearing the whole form after one mistake makes an agent start again, and some won't.

Tie each message to its field

An error floating near a form isn't enough. Agents, like screen readers, need to know which field it belongs to. Two attributes do that:

<label for="zip">ZIP code</label>
<input id="zip" name="zip" autocomplete="postal-code"
       aria-invalid="true" aria-describedby="zip-error">
<p id="zip-error">Enter a 5-digit ZIP code, like 10001</p>

• aria-invalid="true" marks the field as having a problem. Remove it once the value is fixed.
• aria-describedby links the field to its message, so the message is read as part of the field's description.
• Put the message in the page as text next to the field, not in a tooltip that only appears on hover.
• For required fields, use the required attribute, and the right type and autocomplete values, so agents can get it right first time. Labels matter here too: see buttons, links and forms agents can use.

Summarize errors and announce changes

On a long form, such as checkout or account sign-up, add a short summary at the top when it's submitted with problems. It lists each error as a link to its field, and gets keyboard focus so it's the first thing read.

<div role="alert" tabindex="-1" id="error-summary">
  <h2>There are 2 problems with your details</h2>
  <ul>
    <li><a href="#zip">Enter a 5-digit ZIP code, like 10001</a></li>
    <li><a href="#phone">Enter a phone number with area code</a></li>
  </ul>
</div>

For messages that appear without a page load, such as "Promo code applied" or "Only 2 left, quantity updated", use a live region. role="alert" is for urgent errors. role="status" or aria-live="polite" is for confirmations. Keep the message on screen until the next action: toasts that fade after three seconds are easy for an agent to miss.

Make error pages say what happened

Errors aren't only in forms. When an agent follows an old link or a product is discontinued, your error page is what it reads. It should explain, and offer a way forward.

• Not found pages should say the page doesn't exist, and include a search box and links to main categories, so an agent can recover.
• Discontinued products are better served by a page that says so, with links to alternatives, or a permanent redirect to the closest replacement.
• Out-of-stock products should keep their page, show "Out of stock" in text, and set availability to OutOfStock in your product data. Don't turn them into errors.
• Server errors and maintenance pages should say it's temporary and, if you can, when to try again.

Send the status code that matches

Agents and crawlers read your HTTP status code before they read a word of the page. If they disagree, the code usually wins. A "Page not found" message sent with a 200 OK status, often called a soft 404, tells crawlers the page is fine, so the dead page may stay in AI search answers. A real product page that returns an error tells them it's gone.

Two common mistakes: redirecting every missing page to the home page, which leaves agents on a page that doesn't answer the question, and serving a block page with 200, which looks like content. The meanings of these codes are defined in RFC 9110. For 429s and agents, see rate limits for AI agents.

For form submissions, what matters most is that the page sent back shows the errors in text, tied to their fields. Whether you re-render the form with 200 or use 422 is less important than making the message clear.

What AgentScore and Ghost Agents show

AgentScore doesn't have a check for error messages, since it never submits your forms. Several checks cover the groundwork. "Form fields are labelled" checks the home page's fields, and "Cart and checkout fields are labelled for agents" checks the cart and checkout. "AI agents can open your product, pricing and cart pages" compares the status codes AI assistants get with what a browser gets, which catches bot protection sending agents errors that people never see.

To see what happens when an agent actually makes a mistake, run a journey. A Ghost Agent types the test details you give it and records each step, so if a form rejects them without saying why, the replay shows exactly where it stalled. Ghost Agents stop at the payment form, so payment errors are out of scope. To find the errors agents hit in real traffic, see finding the errors AI agents hit in your logs.

A quick checklist

1. Every error message says what's wrong and how to fix it, in text.
2. Each field with an error has aria-invalid="true" and aria-describedby pointing to its message.
3. Long forms show an error summary that links to each field.
4. Messages that appear without a page load use a live region and stay on screen.
5. Entered values are kept after an error.
6. Missing pages return 404 or 410, with a search box and useful links.
7. Out-of-stock products keep their page, with a 200 status.
8. Maintenance and rate-limit responses use 503 or 429 with Retry-After.

For more on the forms themselves, see how to make checkout work for AI shopping agents and sign-up, booking and lead forms AI agents can finish.

Why the cart matters more for agents

A person glances at a cart and knows whether it looks right. An agent has to read it. Before it hands over for payment, a careful agent reports back: what's in the cart, how many, what it costs and when it arrives. It gets every one of those facts from your cart page. If the cart can't be read, the agent can't confirm, and the order stalls at the point where you've already done the hard work of winning it.

For the whole journey from search to payment, see how to make checkout work for AI shopping agents. This guide zooms in on the cart itself.

Quantity controls

The most common problem is a pair of "−" and "+" icons with no names, next to a number that's styled text rather than a field. A person sees a stepper. An agent sees two unnamed buttons and a number it can't change directly.

• Use a labelled number field. A real <input type="number"> whose label names the item lets an agent type the quantity it wants.
• Name the stepper buttons. If you keep plus and minus buttons, give each a name: "Increase quantity of Ethiopia Yirgacheffe".
• Say limits in words. "Limit 4 per order", next to the field. An agent that asks for 6 and silently gets 4 will tell the person the wrong thing.
• Confirm every change. Update the line total and the cart total on the page, and announce it in a status message, so the agent knows the change took.
• Make "Update cart" a real button if your cart needs one, placed next to the quantities, not hidden at the bottom.

A cart line that does all of this:

<li>
  <a href="/products/ethiopia-yirgacheffe?variant=340g">Ethiopia Yirgacheffe, 340 g</a>
  <label for="qty-1">Quantity</label>
  <input id="qty-1" name="qty-1" type="number" min="1" max="4" value="2">
  <span>Limit 4 per order</span>
  <span>$36.00</span>
  <button type="button" aria-label="Remove Ethiopia Yirgacheffe, 340 g">
    <svg aria-hidden="true">…</svg>
  </button>
</li>
<p role="status">Quantity updated to 2. Subtotal $36.00.</p>

Remove buttons

• Name what each one removes. On a cart with three items, three buttons called "×", or nothing at all, leave the agent guessing which is which. "Remove Ethiopia Yirgacheffe, 340 g" leaves no doubt.
• Confirm in text. "Ethiopia Yirgacheffe removed. Undo." A line that simply vanishes could mean it was removed, or that the page is still loading.
• Don't make zero the only way out. Setting the quantity to 0 should work, but a clear remove button shouldn't be missing.

AgentScore's "Buttons and links have names" check looks for exactly this kind of icon-only control on your home page. Apply the same rule in the cart, where it matters more.

Promo code fields

People often ask an assistant to use a code they already have. That only works if the agent can find the field, apply the code and tell whether it worked.

• Give the field a visible label, such as "Discount code", not just placeholder text. Placeholders vanish as soon as typing starts, and many agents ignore them.
• Pair it with a real "Apply" button. Not a form that only submits when someone presses Enter.
• If it's collapsed behind "Have a code?", make that a real button that says it opens something (aria-expanded), not a styled line of text.
• Explain failures in words. "SPRING10 expired on March 31" or "This code doesn't apply to sale items" lets the agent tell the person why. "Invalid code" in red does not. See error messages AI agents can understand.
• Show the applied discount as its own line, with the code name, the amount and a named button to remove it.

Totals the agent can repeat back

The goal is simple: an agent should be able to say "The total is $44.18, including $5.00 shipping and $3.18 tax" and be right. That means each part of the total is on the page as text.

• Show the currency clearly. "$" alone is ambiguous on a site that sells in several countries. See international sites for AI agents.
• Put free-shipping thresholds in words: "Spend $12.00 more for free shipping." A progress bar alone tells an agent nothing.
• Keep prices as text in the page, not in images or drawn on a canvas. See prices AI agents can read.

Carts that survive the handoff

Most agents hand back to the person to sign in or pay, and that can happen hours later, or on another device. A cart that empties in the meantime undoes the agent's work.

• Keep carts for days, not minutes. Don't clear them on a short session timeout.
• Keep the cart at a stable URL, such as /cart, as a full page as well as any slide-out drawer.
• Consider cart links. Some platforms can rebuild a cart from a URL (Shopify calls these cart permalinks), which lets an agent hand the person a link that opens the same cart. Ask your platform what it supports.
• Explain changes. If stock or a price changes while the cart waits, say so on the page: "Only 1 left. We've changed your quantity to 1."

Upsell pop-ups and cart drawers

An upsell window after "Add to cart", or on the way to checkout, is one more thing an agent has to get past.

• If you use one, make it a proper dialog with a heading, a "No thanks" button with text, and a named close button. See cookie banners and pop-ups.
• Never pre-add extras, such as a pre-ticked protection plan. An agent may not notice, and the person pays for something they didn't ask for. In some markets, such as the EU, consumer law already restricts pre-ticked extras.
• Recommendations under the cart are fine. Keep the checkout button above them.
• Give a cart drawer a labelled close button and a real link to the full cart page.

What AgentScore checks on your cart

The free AgentScore scan is read-only: it loads pages but never adds anything to a cart. On many stores the cart is empty when it looks, and the scan says so rather than guessing. For stores, these checks cover the cart:

• AI agents can open your product, pricing and cart pages. It requests your cart as ChatGPT-User, Claude-User and Perplexity-User and compares what they get with what a browser gets.
• AI agents aren't blocked or shown a CAPTCHA at the cart and checkout.
• Cart and checkout fields are labelled for agents. Labels rather than placeholders, the right input types, and autocomplete. If the empty cart shows no fields, this is marked "not tested".
• Shoppers can check out without an account. It reads the cart for wording like "Sign in to check out" or "Check out as a guest".
• Key pages are linked from the home page, including the cart, with ordinary links in the HTML.

Changing quantities, applying codes and reading totals need items in the cart, so they belong to Task completion, which the free scan marks "not tested". To test them, set up a Ghost Agent test with a goal in your own words, such as "Add two bags of decaf, apply the code TEST10 and go to checkout", using a test code you've created, and a pass condition that looks for the code's name on the final page. Ghost Agents stop at the payment form, so no order is placed. See how to test your key journeys with AI agents.

Why after-sale tasks matter

"Where is my order?" is one of the most common reasons customers contact an online store. When an assistant can answer it from your site, that's a support ticket that never gets written. When it can't, the person contacts you anyway, or the assistant pieces together an answer from a carrier's site and gets it wrong.

Returns matter before the sale, too. Assistants read return policies when comparing stores, and a clear, easy returns process is part of what makes a store safe to recommend. See policy pages AI assistants can quote.

Order tracking without a login

An agent can't sign in as the person, but it can usually be given an order number and an email address. That's enough for a well-built order lookup.

• Give order lookup its own page at a stable URL, such as /orders/lookup, linked from your footer and help pages with an ordinary link.
• Ask for two things: the order number and the email address used at checkout. Label both fields, and say where to find the number: "It starts with NW- and is in your confirmation email."
• Show the status in words. "Shipped on October 3 with UPS. Expected Tuesday, October 7", the tracking number as a link, and the items in the order. A progress bar alone tells an agent nothing.
• Put a direct link in every order email that opens the order's status page without signing in. Many platforms offer one. Agents reading an email for the person can follow it straight to the answer.
• Keep failures helpful but safe. "We couldn't find an order matching those details" is right. Don't say which of the two was wrong, which would let anyone test email addresses.
• Limit guessing with rate limits, not a CAPTCHA on every lookup. Answer too many attempts with a 429 status. See rate limits for AI agents.

A lookup form agents can fill in:

<form action="/orders/lookup" method="post">
  <label for="order">Order number</label>
  <input id="order" name="order" autocomplete="off" aria-describedby="order-hint">
  <p id="order-hint">Starts with NW-. You'll find it in your confirmation email.</p>
  <label for="email">Email used at checkout</label>
  <input id="email" name="email" type="email" autocomplete="email">
  <button type="submit">Find my order</button>
</form>

Show only what's needed on a page reached this way: status, items, carrier and the delivery city. Leave out the full address, phone number and payment details.

Returns and exchanges

• Start from a plain-text policy: how long customers have, what condition items must be in, who pays return shipping, how long refunds take and what can't be returned.
• Open the returns portal with an order number and email, the same as tracking. If returns need an account, most agents stop at the door.
• Label every choice. Items as checkboxes named with the product and size, the reason as a list of options in words, and refund, exchange or store credit as radio buttons that say what each one means: "Store credit: issued as soon as we receive the item."
• Explain why an item isn't eligible. "This item was delivered 45 days ago. Returns close after 30 days." See error messages AI agents can understand.
• Summarize before submitting. A review step ("You're returning 1 item for a $64.00 refund to your original payment method") lets the agent check with the person first.
• Say what happens next in words. Where the label is (a link to the PDF, and "We've emailed it to you"), drop-off options, and when the refund will arrive.

Reorders

"Order the same again" usually needs order history, and order history usually needs an account. You can still make it easy:

• Add a "Buy again" button to the order status page, the one reached from the email link. Name it after the item: "Buy again: Ethiopia Yirgacheffe, 340 g".
• Put the same item back in the cart: the same size, color and quantity. Say if anything changed: "The price is now $19.00" or "This size is out of stock."
• Keep product URLs stable. A product link in a year-old confirmation email should still work, or redirect to the replacement product with a note saying so. See canonical URLs and duplicate pages.
• Make subscription controls clear if you sell subscriptions. "Skip next delivery" and "Change quantity" as named buttons, with the next delivery date in words.

Security, accounts and handing back to the person

These tasks touch personal details and money, so there's a real tension. Agents shouldn't need the person's password, and anyone holding an order number shouldn't be able to change an order. The answer is to match the check to the risk:

• Use step-up checks for risky actions only. A one-time code sent by email or text is a clean handoff point: the agent stops and says "Northwind has sent a code to your email", and the person takes it from there.
• Make the handoff obvious. Browser agents typically hand control back for sign-in and payment. A sign-in form with labelled fields, and support for passkeys or password managers, makes that moment quick for the person.
• Never ask for card details to prove who someone is. No one, human or agent, should be typing card numbers to see an order status.
• Keep help content public. Delivery times, return policy and FAQs don't need a login. See login walls and gated content.

The same thinking applies to buying: see guest checkout and sign-up and booking forms agents can use.

How to test these journeys

1. Try it yourself with an assistant. Place a test order, then ask an assistant that can browse to find its status using the order number and email. Watch where it stalls.
2. Check the basics with AgentScore. It checks that AI assistants are let in and can open your key product, pricing and cart pages. It doesn't test order lookup or returns, and the free scan marks Task completion "not tested".
3. Monitor them with Ghost Agents. Describe the journey in your own words, such as "Find the order tracking page, look up order NW-10422 with test@northwind.example, and report the delivery status", and set a pass condition that looks for text like "Shipped" on the final page. For returns, write the goal so the agent stops at the review step, without submitting.

Four rules for the report

• Trend over level. A score of 68 means little on its own. A score that went from 52 to 68 in a quarter, with the reasons, means a lot.
• Keep two questions apart. "Can AI agents use our site?" (readiness, errors, test results) and "Is it turning into business?" (referrals and orders). The first leads; the second follows.
• One owner per measure. Each number needs someone who explains it and acts on it. See who owns agent readiness.
• Note what changed. Releases, bot protection changes, new apps and campaigns. Most movements in these numbers trace back to one of them.

The six measures

In Ghost Agent Labs, the site's Overview page brings several of these together. The sections below name the page that holds each one.

1. AgentScore

Report: the overall score and its band, the scores for Access, Readability and Navigability, and how many findings you fixed and how many are new. The free scan doesn't test Task completion yet, so the score covers the other three categories.

Where: the Readiness page. Every scan is saved, so you can track the score over time, and "All findings" exports to CSV for your developers.

Good looks like: a steady or rising score, no critical findings open for more than a month, and every drop explained. Scan on the same day each month and after major releases, so the numbers compare like with like. The bands are 80 and up Good, 60 to 79 Fair, 40 to 59 Weak and below 40 Poor. See reading your AgentScore report.

2. Agent visits by type

Report: requests by visitor type (search crawlers, AI training crawlers, AI assistants and autonomous agents) compared with last month, the pages agents read most, and any new agents.

Where: the Agent traffic page ("Requests by visitor type", "Agent sessions" and "Pages agents request most"), and "What changed" on the Overview, which lists first-time agents and agents whose traffic rose or fell by half or more.

Good looks like: AI assistant and AI search traffic holding steady or growing, and agents reaching product, pricing and policy pages rather than stalling on the home page or old URLs. Don't celebrate total volume. A busy training crawler isn't a customer. Assistant requests are the closest thing to a customer visit from AI, because each one is usually a person's question. See how to measure AI agent traffic.

3. Verification share

Report: of the requests claiming to be a known agent, the share that were verified, couldn't be verified, and were spoofed.

Where: the Verification & spoofing page.

Good looks like: spoofed traffic close to zero, or falling after you block it, and a verified share that rises as more operators publish ways to check their agents. This measure also keeps the rest of the report honest: never report "ChatGPT visited 40,000 times" from user agent names alone, because impostors use those names too. See how to tell if an AI crawler is real.

4. Blocks and errors

Report: the share of agent requests that were blocked (401, 403 and 429 responses), the 404 and server error rates, and the agents and pages most affected.

Where: "Errors: agents vs people" on the Overview, "What agents got back" on the Agent traffic page, and the status codes on each agent's page.

Good looks like: blocked responses to verified AI assistants near zero, agents getting errors about as often as people do rather than much more, and agent 404s trending down as you redirect old URLs. A sudden rise almost always follows a change to bot protection or rate limits. See finding the errors AI agents hit.

5. Ghost Agent pass rates

Report: for each journey you test, such as checkout, product search or sign-up, whether it's passing, flaky or failing, its pass rate, and the step where failures cluster.

Where: the Ghost Agents page, with a replay of every run.

Good looks like: your money journeys passing, flaky results treated as findings rather than noise, and failures fixed within a sprint. A pass rate below 100% means some customers' assistants would fail too. This is the measure closest to revenue, because it tests whether an agent can do the job, not just whether the page looks right. See how to test your key journeys with AI agents.

6. AI referral visits and orders

Report: visitors who arrived from AI assistants' answers, by assistant, their conversion rate, and orders and revenue from AI.

Where: your analytics, set up as in tracking visits and sales from AI assistants. In Ghost Agent Labs, "Visitors from AI assistants" is on the Overview. Send your store's orders to the site's orders URL and Mission Control shows "Agent-driven revenue": orders referred by an AI answer or placed by an agent, and their share of revenue.

Good looks like: growth from whatever base you start at, and conversion from AI referrals that holds up against your other referral sources. Present it as a floor, not a total: app traffic and copied links arrive as "direct", and many AI answers influence a purchase without a click.

Reading the measures together

Single numbers mislead. These combinations are where the report earns its keep:

• Score steady, blocks up. Something changed in bot protection or rate limits. Check the agent's page for which pages are affected and when it started.
• Visits up, referrals flat. Assistants are reading you but not sending people. Look at product data, prices and policy pages: are they giving assistants what they need to recommend you?
• Score unchanged, pass rate down. A journey broke in a way the scan can't see: a new pop-up, a redesigned size picker, a checkout app. The run replay shows where.
• Referrals up, checkout test failing. AI is sending you customers and losing some of them at the last step. This is the most urgent combination on the page.
• Spoofed share up. Scrapers are using agent names. It doesn't hurt readiness, but it inflates traffic numbers and costs bandwidth.

A one-page template

1. Headline. One sentence: better, worse or the same, and why.
2. Scorecard. The six measures, this month against last month, with an arrow and a short note each.
3. What changed. Site releases, bot protection changes and new agents, linked to the numbers they moved.
4. Fixed this month. The findings closed and the journeys repaired.
5. Next month. The top three fixes, each with an owner.
6. Decisions needed. For example, whether to allow a new AI agent, or budget for platform work.

It builds on how to measure AI agent traffic, which covers getting hold of your logs in the first place.

The errors that matter

Challenge pages are the easiest to miss, because some bot protection serves them with a 200 status, so they look like successes. Look for your CDN's security or bot action field instead. Cloudflare, for example, adds a cf-mitigated: challenge header to challenge responses. A response that's much smaller than the real page usually is one too.

What your logs need

Most logs already have the basics. Check yours include:

• Time, client IP address and user agent. The IP address is what lets you verify the agent later.
• Path and status code. Query strings can be dropped; you rarely need them, and they can hold personal data.
• Response size and response time. These catch challenge pages and timeouts.
• The CDN's security action, if it has one: allowed, blocked, challenged, rate limited.

On nginx, a log format like this captures them:

log_format agents '$time_iso8601 $remote_addr "$request_method $uri" '
                  '$status $body_bytes_sent $request_time "$http_user_agent"';

If you only have the standard "combined" access log, this one-liner counts status codes for three AI assistants:

grep -E 'ChatGPT-User|Claude-User|Perplexity-User' access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn

Example queries

Once logs are in a database or log tool, a few queries answer most questions. The examples below are illustrative, written for PostgreSQL against a table called requests with columns ts, client_ip, user_agent, path, status, bytes, duration_ms and security_action. Rename them to match your own log store; the same ideas work in BigQuery, Athena or a log search tool.

Error rates by agent, last 7 days

SELECT
  CASE
    WHEN user_agent ILIKE '%ChatGPT-User%'    THEN 'ChatGPT-User'
    WHEN user_agent ILIKE '%OAI-SearchBot%'   THEN 'OAI-SearchBot'
    WHEN user_agent ILIKE '%Claude-User%'     THEN 'Claude-User'
    WHEN user_agent ILIKE '%Perplexity-User%' THEN 'Perplexity-User'
    WHEN user_agent ILIKE '%PerplexityBot%'   THEN 'PerplexityBot'
    WHEN user_agent ILIKE '%Googlebot%'       THEN 'Googlebot'
    ELSE 'everything else'
  END AS agent,
  COUNT(*) AS requests,
  COUNT(*) FILTER (WHERE status IN (401, 403)) AS blocked,
  COUNT(*) FILTER (WHERE status = 429)         AS rate_limited,
  COUNT(*) FILTER (WHERE status = 404)         AS not_found,
  COUNT(*) FILTER (WHERE status >= 500)        AS server_errors,
  ROUND(100.0 * COUNT(*) FILTER (WHERE status >= 400) / COUNT(*), 1) AS error_pct
FROM requests
WHERE ts >= now() - interval '7 days'
GROUP BY 1
ORDER BY requests DESC;

"Everything else" mixes people with other bots, many of them scanners probing for pages that don't exist. For a fair baseline, compare with requests from ordinary browsers: if AI agents get errors far more often than people do, something is turning them away.

Where AI assistants are turned away

SELECT path, status, COUNT(*) AS hits
FROM requests
WHERE ts >= now() - interval '7 days'
  AND user_agent ~* '(ChatGPT-User|Claude-User|Perplexity-User)'
  AND (status IN (401, 403, 429) OR security_action = 'challenge')
GROUP BY path, status
ORDER BY hits DESC
LIMIT 20;

Challenges and timeouts by day

SELECT date_trunc('day', ts) AS day,
  COUNT(*) FILTER (WHERE security_action = 'challenge')   AS challenged,
  COUNT(*) FILTER (WHERE status IN (499, 504))            AS timed_out,
  COUNT(*) FILTER (WHERE duration_ms > 10000)             AS over_10s
FROM requests
WHERE user_agent ~* '(ChatGPT-User|Claude-User|Perplexity-User|OAI-SearchBot)'
GROUP BY 1
ORDER BY 1;

A step change on one day usually lines up with a release or a bot protection change. That's your first lead.

Separating verified agents from claimed ones

Everything above trusts the user agent, and a user agent is only a claim. Scrapers often call themselves ChatGPT-User or Googlebot to get past bot protection. When your firewall blocks them, that 403 is working as intended. So before you loosen any rule, split agent errors into three groups:

• Verified. The request came from the operator's published IP ranges, passed forward-confirmed reverse DNS, or carried a valid signature (Web Bot Auth, still an emerging standard). Errors here are real problems to fix.
• Spoofed. It used a known agent's name but failed the operator's check. Blocking these is correct.
• Unverifiable. The operator doesn't publish a way to check. Decide case by case.

Operators such as OpenAI publish their agents' IP ranges (see OpenAI's crawler documentation). Load them into a table, refresh it regularly, and join:

-- agent_ranges(agent text, cidr cidr), loaded from each operator's published list
SELECT r.agent IS NOT NULL AS verified, l.status, COUNT(*) AS hits
FROM requests l
LEFT JOIN agent_ranges r
  ON l.client_ip::inet << r.cidr
 AND l.user_agent ILIKE '%' || r.agent || '%'
WHERE l.user_agent ~* '(ChatGPT-User|OAI-SearchBot)'
  AND l.ts >= now() - interval '7 days'
GROUP BY 1, 2
ORDER BY 1 DESC, hits DESC;

For the methods in detail, including reverse DNS and signed requests, see how to tell if an AI crawler is real.

From errors to fixes

• Verified assistants get 403s on product or policy pages: allow verified bots in your bot protection. See bot protection and CAPTCHAs and setting up your CDN for AI agents.
• Repeated 429s: give verified agents their own limits, and send Retry-After. See rate limits for AI agents.
• Blocks only from certain countries or networks: see geo-blocking, VPN blocks and AI agents.
• Challenges at the cart or checkout: these cost orders directly. AgentScore checks for them, and the checkout guide covers the fix.
• The same 404s week after week: redirect old URLs to their current pages.
• Server errors and timeouts: cache the pages agents request most, and check them in your monitoring like any other.

If writing queries isn't your team's idea of a good week, Ghost Agent Labs does this from your Cloudflare, Vercel or JSON logs. The Agent traffic page shows what agents got back, each agent's page shows its status codes, the Verification & spoofing page splits verified from spoofed, and Alerts tell you when a major AI assistant starts being blocked. Either way, check after every bot protection or CDN change, and at least monthly as part of your agent readiness KPIs.

Two ways an agent can use your site

Today, an AI agent asked to find a waterproof jacket in a medium on your store has to work your pages. It loads the home page, finds the search box, reads the results, opens a product, works out the size picker, and reads the price. Every step depends on your page layout. A redesigned menu or a new pop-up can stop it.

The alternative is to tell the agent what it can do and let it ask directly. Instead of driving the search box, it calls a search_products action with "waterproof jacket, medium" and gets back a clean list of products, prices and stock. That's the idea behind both MCP and WebMCP.

• Faster. One request instead of a dozen page loads and clicks.
• More reliable. Actions don't change when you redesign the page.
• More accurate. You decide exactly what data comes back, so agents quote the right price and stock.

It doesn't replace a usable website. Many agents will keep using your pages for a long time, and the people they hand back to always will. Think of MCP and WebMCP as an extra, faster lane for the agents that support them.

What MCP is

The Model Context Protocol (MCP) is an open protocol for connecting AI applications to outside tools and data. Anthropic introduced it in late 2024, and it has since been adopted by many AI assistants, developer tools and software companies.

An MCP server is a small service that describes a set of tools (actions an agent can call, each with a name, a plain-language description and the inputs it expects) and can also offer resources (data it can read, such as a catalog or a policy). An AI application connects to the server, reads the list, and calls the tools it needs to complete a person's request.

For a website, a remote MCP server runs alongside your site, usually on your own domain, and talks to the same systems your site does. Typical tools for an online store:

Software companies might offer tools to look up plans and pricing or search documentation. Service businesses might offer tools to check availability and request a booking.

Commerce platforms are starting to offer MCP servers for the stores they host, so check with yours before building one. If you have a developer team and an existing API, a read-only server is often a small project, because it wraps calls you already make.

What WebMCP is

WebMCP is an early proposal for bringing the same idea into the web page itself. Instead of running a separate server, your page registers tools with the browser through a new JavaScript API, navigator.modelContext. An agent working in that browser can then call the page's tools directly instead of clicking through it.

It's being developed in the open at the W3C's Web Machine Learning Community Group, with engineers from Google and Microsoft among those working on it. Community group work is incubation, not a finished standard, and browsers don't ship WebMCP by default yet. Expect the details to change.

Two features make it interesting for websites:

• It reuses your page code. The tool can call the same functions your search box or add-to-cart button already calls.
• It works with the person's session. Because it runs in the page, it can use the cart and sign-in the person already has, without separate API credentials.

A tool registered in a page looks roughly like this, guarded so it does nothing in browsers without the API:

if ("modelContext" in navigator) {
  navigator.modelContext.registerTool({
    name: "search_products",
    description: "Search Northwind Outdoor products by keyword, size and maximum price.",
    inputSchema: {
      type: "object",
      properties: {
        query: { type: "string" },
        size: { type: "string" },
        maxPrice: { type: "number" }
      },
      required: ["query"]
    },
    async execute({ query, size, maxPrice }) {
      const results = await searchCatalog({ query, size, maxPrice }); // your existing search
      return { content: [{ type: "text", text: JSON.stringify(results) }] };
    }
  });
}

The proposal also describes a declarative form: adding a toolname attribute (with a description) to an existing <form>, so a search or contact form becomes a tool without new JavaScript. Attribute names may change as the proposal develops.

MCP or WebMCP?

For most businesses, an MCP server is the practical choice today, and WebMCP is worth a small experiment so you're ready if browsers adopt it.

What AgentScore looks for

The AgentScore check "Agents can use your site through MCP or WebMCP" is part of the Access category. It passes if it finds any of these:

• An MCP server description at /.well-known/mcp.json or /.well-known/mcp. There's no settled standard for advertising an MCP server yet, and proposals for a well-known server description are still being discussed, so we look in the most likely places.
• A link in your home page: a <link> or <a> whose rel or type mentions MCP.
• An MCP server listed in your llms.txt, with its URL. See how to write an llms.txt file.
• WebMCP tools registered in your home page. Because browsers don't ship WebMCP yet, the scanner provides a stand-in navigator.modelContext before your scripts run, so pages that check for the API register their tools as they would in a supporting browser. We record the tool names.
• Forms marked up as WebMCP tools with a toolname attribute.

If none is found, the check is a warning rather than a failure. Agents can still use your pages; they just have to do it the slow way. It carries less weight than the checks that decide whether agents can get in at all, such as bot protection and robots.txt.

How to start, safely

1. Ask your platform first. If your commerce platform or CMS offers an MCP server or WebMCP support, turning it on is far cheaper than building one.
2. Start read-only. Search, product details, pricing, stock and policies. These answer most agent questions and can't change anything.
3. Write tool descriptions for a reader who has never seen your site. The description is how an agent decides which tool to call. "Search products by keyword, size and maximum price; returns name, price, stock and URL" beats "Search".
4. Return the same facts your pages show. Prices, stock and policies must match the website exactly, or agents will quote figures your checkout won't honor.
5. Keep people in charge of money. For anything that changes state, such as building a cart, return a link the person opens to review and pay. Leave fully agent-led checkout to the agentic commerce protocols designed for it.
6. Protect it like an API. Rate limits, logging and authorization for anything personal. See rate limits for AI agents.
7. Advertise it. List the server in llms.txt with a link to its documentation, and add a well-known description or a <link> in your page head.

What it means for the business

An MCP server or WebMCP tools won't bring traffic on their own. They make each agent visit more likely to end well: the right product found, the right price quoted, the cart built. Because support among AI applications is still growing, the best order of work is the usual one. Make sure agents can get in and read your pages first, then add a direct lane for the agents that can use it.

Not sure where you stand on the basics? Start with what agent readiness means, then run AgentScore to see every Access check, including this one.

Why protocols, when agents can use checkout pages?

A browser agent can shop through your website, and for now that's how most agents buy. We cover that journey in how to make checkout work for AI shopping agents. But checkout pages were built for people. An agent filling them in has to guess at fields, cope with pop-ups and CAPTCHAs, and usually hand back to the person to pay.

A protocol replaces the page-filling with a structured conversation between the assistant and your store:

• The assistant asks for a product, variant, quantity and delivery address.
• Your store answers with the real total, shipping options, tax and any errors, in data rather than on a page.
• The person approves the purchase inside the assistant.
• Payment is passed in a form that proves the person authorized it, without the assistant seeing raw card numbers.
• Your store creates the order and stays responsible for fulfillment, returns and customer service.

The protocols you'll hear about

This is a young and fast-moving area. The descriptions below are deliberately brief, and the protocols' own documentation is the place for detail. Programs, eligibility and regions change often.

Card networks and payment providers have also announced their own programs for payments made by agents. Most of these sit underneath the protocols above, at the payment layer, rather than competing with them.

What it means for your store

• Assistants become a sales channel. Supporting a protocol can let a person buy from you without leaving the assistant they asked. That puts you in front of shoppers at the moment they decide.
• You stay the merchant. In the protocols announced so far, the store keeps the order, the customer relationship and responsibility for fulfillment. Check the terms of any program you join.
• Your data has to be right. An assistant will quote the price, stock and delivery date your feed or API returns. Mistakes become disappointed customers at scale. See product data AI shopping agents can read.
• Fraud and risk rules need a second look. Orders arrive without the usual browser signals. Ask your payment provider how it scores them.
• Returns and support still come to you. Make sure your team can see which orders came through an assistant.

How stores get support

Very few stores will build a protocol integration themselves. Support usually arrives in one of three ways:

1. Through your commerce platform. Hosted platforms are adding protocol support for the stores they run, sometimes as a setting you switch on.
2. Through your payment provider. Some providers handle the agent-facing checkout and payment for you.
3. Through an AI platform's merchant program. Some assistants ask merchants to apply, and review product data and policies before listing them.

If you run a custom stack, the specifications are public, but budget for real engineering: order creation, tax, shipping, inventory and payment all have to behave exactly as they do on your site.

Questions to ask your platform and payment provider

• Which agentic commerce protocols do you support today, and which are planned?
• Is it on by default, or do we need to enable it or apply?
• Which product data does it use, and how do we keep it in sync with the website?
• How are agent orders marked in our order system and analytics?
• How are fraud screening, chargebacks and refunds handled for agent orders?
• Can we limit it to certain products, countries or order values while we learn?

What AgentScore checks

For stores, AgentScore includes the Access check "Agents can check out through an agentic commerce protocol". It's skipped for sites that aren't stores.

It can only see what a store publishes openly. Today that means a Universal Commerce Protocol profile: the scanner requests /.well-known/ucp, and if it gets back a JSON document, the check passes.

The Agentic Commerce Protocol is different. Integrations are set up privately between each store and each AI platform, so there's nothing on your site for a scanner to find. If you support ACP but not UCP, this check will still show a warning. That's a limit of what can be seen from outside, not a problem with your store. The advice in the report names both protocols.

A warning here costs little. The check has a small weight in the Access category, because most agents still shop through the website, and that's where the checks that decide whether agents can get in and get through checkout matter more. Those include bot protection, CAPTCHAs at the cart and checkout, and guest checkout.

Where to start

1. Fix the website journey first. Make sure agents can get in, read your product pages and reach checkout. Every protocol relies on the same product data and policies.
2. Get your product data in order. Accurate prices, variants, stock and delivery times in structured data and in any product feeds you publish.
3. Ask the questions above of your platform and payment provider, and turn on what's available.
4. Watch how agents arrive. Track agent visits and the orders that come from assistants, so you can tell whether a protocol is earning its keep. See how to measure AI agent traffic and tracking visits and sales from AI assistants.

Agentic commerce protocols will keep changing for a while. The stores best placed to benefit are the ones whose websites, data and policies already work for agents, because that's what every protocol is built on.

Why accounts stop agents

A forced account is a known source of abandoned checkouts with people. With AI agents it's usually worse, for practical reasons:

• Agents can't sign up on someone's behalf. Creating an account means choosing a password, agreeing to terms and often confirming an email address. Well-behaved agents won't invent a password or accept terms for a person, and they can't open the person's inbox to click a confirmation link.
• Signing in is a handoff. Browser agents generally stop at sign-in and pass control back to the person. Some people will finish; many won't, especially if they asked the assistant precisely so they wouldn't have to.
• Sign-in pages attract bot defenses. Login forms are a favorite target for credential-stuffing attacks, so they often carry the strictest CAPTCHAs and bot rules. An agent sent there meets the hardest wall on your site.
• The failure happens at the last step. The agent has already done the expensive work: searching, comparing, choosing. Losing the order at sign-in wastes all of it, and the assistant may recommend a store where it could finish instead.

Guest checkout fixes all four. The agent, or the person it hands back to, enters an email address and a delivery address and carries on to payment.

What good guest checkout looks like

• Guest is a clear choice, in words. A button or link that says "Check out as guest" or "Continue as guest", not an unlabelled icon or a link hidden below a sign-in form.
• Guest is the default path, or at least equal to sign-in. Many stores go straight to the contact and delivery form and offer sign-in as a link at the top.
• It asks only for what the order needs: email, name, delivery address, phone if your carrier requires it. Every field is labelled and uses the right autocomplete value. See buttons, links and forms agents can use.
• No password field on the guest path. If you want to offer an account, offer it after the order.
• No CAPTCHA on every checkout. Use risk-based checks that challenge only suspicious orders. See bot protection and CAPTCHAs.
• The cart doesn't say otherwise. A cart page that reads "Sign in to check out" undoes a perfectly good guest checkout behind it.

Keeping the benefits of accounts

Accounts matter to most stores: repeat purchases, order history, loyalty and marketing consent. Guest checkout doesn't mean giving those up. It means moving the ask to a moment when it doesn't cost you the sale.

Some stores genuinely need accounts: trade-only wholesale, age-restricted products, prescription items, or subscription services where the account is the product. If that's you, say so clearly on product pages and in your llms.txt, so agents can tell the person up front rather than discovering it at checkout.

How AgentScore checks for guest checkout

The Access check "Shoppers can check out without an account" applies to stores. The scanner is read-only: it loads your cart and checkout pages, but never submits a form or adds anything to a cart. It looks at what a shopper would see:

• Pass if the cart or checkout offers a guest option in words, such as "Check out as guest", "Continue as guest" or "No account needed", or if the checkout asks for contact and delivery details with no password field.
• Fail if the checkout redirects to a sign-in page, asks for a password with no guest option, or the cart says shoppers must sign in to check out.
• Not tested if the checkout only opens with items in the cart. Because the scanner never adds items, many checkouts send it back to an empty cart. In that case the check is skipped and doesn't count against your score, rather than guessing.

It carries a moderate weight in the Access category, alongside the related check that AI agents aren't blocked or shown a CAPTCHA at the cart and checkout. If your checkout couldn't be tested, a Ghost Agent can walk the full journey with items in the cart. See how to test your key journeys with AI agents.

How to check your own store in five minutes

1. Open your store in a private browsing window, so you're signed out.
2. Add any product to the cart and open the cart page. Read it as an agent would: does any text say you must sign in?
3. Press the checkout button. Note where you land: a sign-in page, a choice between sign-in and guest, or straight into a contact and delivery form.
4. If there's a guest option, make sure it's a real button or link with words on it, and that it works with the keyboard alone.
5. Go through to the payment step without creating an account. Note every field, pop-up and challenge on the way.

Guest checkout and agentic commerce

Assistants that buy through an agentic commerce protocol don't use your checkout pages, so they bypass this problem. But most agents still shop through the website, and the people they hand back to always do. Guest checkout is the simplest change that makes both work, and it helps human shoppers in a hurry just as much.

For the rest of the journey, from search to the payment handoff, see how to make checkout work for AI shopping agents. Then run AgentScore to see where your store stands.

Why limits catch good agents

Most rate limits count requests per IP address. That made sense when one address meant one visitor. It works less well for AI agents:

• Agents share addresses. Assistant fetchers and browser agents run in cloud data centers, so many different people's requests can arrive from a small pool of addresses. A per-IP limit treats them as one heavy user.
• Agent visits come in bursts. A person asks an assistant to compare three jackets, and it opens a category page, three product pages and a returns policy within seconds. That's a normal visit, but it can look like a spike.
• Crawlers and assistants get lumped together. A training crawler reading your whole catalog and an assistant answering one shopper's question have very different value to you, but a single "bots" limit treats them the same.
• Blocks look like limits. Many setups answer a rate-limited request with a 403 Forbidden or a challenge page. The agent can't tell it should wait, so it reports your site as unavailable.

Say "slow down" properly: 429 and Retry-After

HTTP has a status code made for this. 429 Too Many Requests, defined in RFC 6585, tells the client it has sent too many requests. Pair it with a Retry-After header, defined in RFC 9110, saying how many seconds to wait:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: text/plain
Too many requests from this client. Please wait 30 seconds and try again.

Well-behaved crawlers and agents treat this as an instruction, not a door slammed shut. Google's crawler documentation, for example, says Googlebot slows down when it gets 429, 500 or 503 responses. Compare the alternatives:

Keep the 429 lightweight: a short plain-text message, no heavy page, and never a CAPTCHA, which agents can't solve. There's also an IETF draft for RateLimit headers that tell clients their remaining allowance before they hit the limit. It's still a draft, so treat it as a bonus for API clients, not a replacement for Retry-After.

Limit by who, not just by address

The fairest limits are set per agent, based on verified identity. Most CDNs and bot management products can verify the major AI agents against their operators' published IP ranges, reverse DNS or signed requests (see how to tell if an AI crawler is real). Once you know who's asking, you can give each kind of visitor a limit that fits its job:

The right numbers depend on your traffic and infrastructure, so we don't suggest specific figures. Start from what your logs show a normal agent visit looks like, then set limits comfortably above it.

Limit actions more than pages

Abuse mostly targets actions: logins, account sign-ups, discount codes, gift card balances, stock checks and search. Pages that only display information are cheap to serve, especially from a cache. Put your tightest limits on:

• Login, sign-up and password reset
• Coupon, gift card and checkout submission endpoints
• Search and filter endpoints that hit your database
• Public APIs and any MCP server you offer

Keep product, category, pricing, policy and help pages loose, and cache them well. Those are the pages agents need to answer questions about you, and the ones that turn into sales.

Other ways to reduce load

• Cache aggressively. A product page served from a CDN cache costs almost nothing, however often it's requested.
• Support conditional requests. ETag and Last-Modified headers let crawlers ask "has this changed?" and get a tiny 304 Not Modified response if not.
• Keep your sitemap accurate. Correct lastmod dates help crawlers revisit only what changed. See XML sitemaps for AI agents.
• Don't rely on Crawl-delay. This robots.txt line is ignored by Google and supported unevenly elsewhere. Rate limits with 429 work for every client.

How to tell if limits are hurting you

1. Check your logs by agent. Look at the share of requests from AI assistants that get 429, 403 or challenge responses. A handful of 429s for a crawler is healthy; repeated 429s or 403s for assistant fetchers mean customers' questions are going unanswered. See how to measure AI agent traffic.
2. Run AgentScore. It requests your home page as a normal browser and as several AI agents, and requests your key pages, cart and checkout as AI assistants. Any error status an agent gets while the browser gets through, including 429, counts as blocked in the bot protection, key pages and checkout checks.
3. Ask your CDN provider whether its rate limiting treats verified bots separately, and whether it returns 429 or 403 when a limit is hit.

Rate limits and bot protection work together. For the wider picture, read bot protection and CAPTCHAs and, for the business decision about which agents to let in at all, should you block AI crawlers?

Why images are a blind spot for agents

An AI agent helping someone shop usually reads your page as text. It fetches the HTML, strips out the layout and works from what's left. Images arrive as file names and URLs, which tell it very little. IMG_4471_final.jpg doesn't say whether the jacket is navy or black.

Some agents do look at the page as a picture, taking screenshots and reading them with a vision model. They can make out a photo, but slowly, at a cost, and not always correctly. Text is faster, cheaper and unambiguous, so even those agents lean on it when it's there.

Alt text is the bridge. It's an attribute on the image tag, written for anyone who can't see the image: screen reader users, people on slow connections, search engines and now AI agents. Writing it well is the same work for all of them. We make that case in Accessibility work is agent readiness work.

What AgentScore checks

The "Images have text descriptions" check is part of the Readability category. It loads your home page in a real browser and looks at the images people can actually see, ignoring tiny ones under 32 pixels wide such as icons and tracking pixels. For each one it asks a simple question: does it have an alt attribute at all?

An empty alt="" counts as described. That's deliberate: it's the correct way to mark an image as decorative, and it tells agents and screen readers to skip it. The check can't judge whether your descriptions are any good, so that part is up to you.

It's a small check by weight, because a missing description rarely stops an agent on its own. But the same gaps tend to show up elsewhere. A linked product image with no alt text is also a link with no name, and that's caught by the "Buttons and links have names" check in Navigability, which carries more weight.

How to write alt text for product images

Describe what a shopper needs to know from the photo, in the words they'd use. Lead with the product, then add the details the image actually shows.

• Name the product and the variant. If the photo shows the navy version, say navy. Agents use this to match images to options.
• Say what's different about each shot. Front, back, detail, in use, on a model. Five images all called "Field jacket" waste the opportunity.
• Include facts the photo proves. Model height and size worn, what's in the box, the scale of an object next to something familiar.
• Keep it short. One short sentence is usually enough. Long specifications belong in the product description, not the image.
• Don't stuff keywords. Agents read alt text as a description. A list of search terms reads as noise and helps nobody.
• Don't start with "image of". Everyone reading it already knows it's an image.

Decorative images, logos and linked images

Not every image needs words. Background textures, divider graphics and lifestyle photos that repeat the headline next to them can take alt="". That's different from leaving alt out: an empty value says "nothing to see here", while a missing one leaves agents and screen readers guessing, and some will read out the file name instead.

Images that are links need extra care, because the alt text becomes the link's name. A product tile where the whole image links to the product page should have alt text that works as a link, such as the product name. Your logo in the header links home, so alt="Northwind home" or simply your brand name works better than alt="logo".

Text baked into images is a common trap on home pages. A banner that says "40% off outerwear this weekend" as pixels is invisible to most agents. Put the offer in real HTML text over the image, or at minimum repeat it in the alt text.

Doing this at catalog scale

A store with thousands of products can't write every description by hand. You don't need to. Most of it can be generated from data you already have.

1. Fix the template first. Make sure your theme outputs an alt attribute on every product image, and falls back to the product name and variant when no custom text is set. That alone gets most stores to a pass.
2. Use the image metadata your platform stores. Shopify, WooCommerce, Magento and most other platforms have an alt text field per image. Fill it on new products as part of the listing checklist.
3. Backfill your bestsellers. Write proper descriptions for the products that earn the most, where shoppers are most likely to ask an assistant about them.
4. Review generated text before publishing. Tools that write alt text with AI are useful for a first draft, but they can describe the wrong color or invent details. Check a sample, especially for variants.
5. Add it to the content workflow. Merchandisers and content teams usually own this, not developers. Make alt text a required field when new images are uploaded.

Where alt text fits

Alt text is the visual part of a bigger job: making sure everything a shopper learns from your page is also available as text. The other parts are prices and stock in a form agents can read (see Prices AI agents can read and Product data that AI shopping agents can read), and content that's in the HTML rather than added by JavaScript (see our guide to JavaScript-only content).

Run AgentScore to see how your home page does, then spot-check a few product pages by hand. For the full picture of what agents need from a site, start with the agent readiness guide.

What a sitemap does for AI agents

An agent arriving at your site has two ways to find things: follow links from the home page, or read a list someone has prepared. Following links is slow and fragile. Menus built with JavaScript, products buried four clicks deep and pages that only appear in search results can all be missed.

A sitemap solves that. It's an XML file, usually at /sitemap.xml, that lists your important URLs. Crawlers from AI search engines use it to decide what to fetch. Agents doing a task can use it to locate a product or policy page directly. And it gives every system the same, complete picture of what you publish.

It doesn't replace good links. A page that's only in your sitemap is still a page most visitors and many agents never reach. Treat the sitemap as the safety net under your navigation, not instead of it. Our guide to linking key pages from the home page covers the other half.

What AgentScore checks

The "Valid sitemap" check, in the Readability category, looks for your sitemap the way crawlers do:

1. It reads Sitemap: lines in your robots.txt.
2. It then tries /sitemap.xml and /sitemap_index.xml.
3. For each, it checks the file loads with a 200 status and is a real sitemap: XML with a <urlset> or <sitemapindex> element.

It passes when it finds one, and reports how many entries it holds and whether it's listed in robots.txt. It fails if none of those places has a valid sitemap. A home page or a "not found" page served at /sitemap.xml doesn't count, which catches a common problem with some site builders.

The sitemap also feeds other checks. AgentScore uses it to find a product or pricing page when the home page doesn't link to one. If the sitemap is an index, it reads product sitemaps first, since most platforms name them that way. That's why a missing sitemap can leave other checks marked "not tested": without a product page to look at, there's nothing to check prices or product data on.

What a good sitemap looks like

The format is defined at sitemaps.org. A minimal one for a small store:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://northwind.example/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
  <url>
    <loc>https://northwind.example/products/ethiopia-yirgacheffe</loc>
    <lastmod>2026-10-07</lastmod>
  </url>
  <url>
    <loc>https://northwind.example/pages/shipping</loc>
    <lastmod>2026-08-14</lastmod>
  </url>
</urlset>

Larger sites split it up. Each sitemap file can hold up to 50,000 URLs, and a sitemap index lists the individual files:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://northwind.example/sitemap_products_1.xml</loc></sitemap>
  <sitemap><loc>https://northwind.example/sitemap_collections_1.xml</loc></sitemap>
  <sitemap><loc>https://northwind.example/sitemap_pages_1.xml</loc></sitemap>
</sitemapindex>

Then tell crawlers where it is, with one line in robots.txt:

Sitemap: https://northwind.example/sitemap.xml

What to include, and what to leave out

The rule of thumb: list the pages you'd be happy for an AI assistant to send a customer to, each at its one true address. If you include a page in the sitemap but block it in robots.txt, you're sending crawlers mixed signals.

Keep it accurate

• Generate it automatically. Shopify, WooCommerce, Magento, BigCommerce, WordPress SEO plugins and most modern frameworks produce a sitemap for you. A hand-made sitemap goes stale the week after it's written.
• Make lastmod honest. It should change when the content does, such as a price or stock change, not every time the file is rebuilt. Google's documentation says it uses lastmod only when it's consistently accurate, and other crawlers are likely to treat it similarly.
• Don't bother with priority and changefreq. They're in the protocol, but Google says it ignores them. Your time is better spent on accurate dates.
• Remove dead pages. Discontinued products that return 404, or redirect to a category, shouldn't stay listed.
• Keep it reachable. Make sure bot protection doesn't challenge crawlers asking for the sitemap. See our guide to bot protection and CAPTCHAs.

Sitemap, llms.txt or both?

Both, because they do different jobs. A sitemap is complete: every page you want found, with no explanation. An llms.txt file is selective: a short, described guide to the pages that matter most, written for AI. A crawler indexing your catalog wants the sitemap. An assistant trying to answer "what's their returns window?" benefits from llms.txt pointing straight at the returns page.

Next steps

1. Confirm your platform generates a sitemap, and that it loads.
2. Add a Sitemap: line to robots.txt if it's missing.
3. Spot-check that your bestselling products and your policy pages are listed.
4. Run AgentScore. The "Valid sitemap" check will show what it found and how many entries it counted.

For how the sitemap fits with the other things agents need, see the agent readiness guide.

Why the labels matter more to agents than to people

A person landing on your page takes in the design at a glance: the hero image, the logo, the layout. An agent sees none of that. It sees text, in order, and leans heavily on the few pieces of text that say what the page is.

• The <title> is the page's name. It shows in browser tabs and search results, and it's often the first thing an agent records about a page it has visited.
• The meta description is a one or two sentence summary. Search engines and AI search tools may show it, quote it, or use it to decide whether a page is worth opening.
• Headings (<h1> to <h6>) are the outline. An agent looking for "delivery times" on a long FAQ uses them to jump to the right section.
• The lang attribute on the <html> tag says what language the page is in, so the agent reads and quotes it correctly.

If you've done SEO work, most of this will be familiar. The difference is that an agent isn't ranking your page among ten others; it's trying to complete a task, and a vague label can mean it picks the wrong page or gives up. There's more on that shift in SEO got you found. Agent readiness gets you chosen.

What AgentScore checks

The "Clear page title, description, and headings" check, in the Readability category, reads your home page's HTML before any JavaScript runs and looks for four things:

All four present is a pass, and the result shows your title. One or two missing is a warning. It fails if there's no title at all, or if three or more are missing. The check confirms these things exist; it can't tell whether the wording is helpful. The rest of this guide is about that part.

Writing titles agents can use

A good title says what the page is and whose it is, specific enough to tell it apart from every other page on your site.

• Put the specific part first. The product or topic, then your brand.
• Make every title unique. If twenty pages share a title, an agent comparing them has nothing to go on.
• Include the variant or size when it matters. "340 g" or "Pro plan" is exactly what an agent needs to match a request.
• Skip the slogans. "Where every cup tells a story" says nothing about what's on the page.

Writing descriptions that summarize the page

Write the meta description as the answer to "what will I find here?" One or two plain sentences: what the page offers, for whom, and anything that sets it apart, such as price range, delivery area or a free trial.

<meta name="description" content="Light-roast single-origin coffee from
Yirgacheffe, Ethiopia, with notes of jasmine and lemon. Roasted to order,
340 g bag, ships free in the US over $40.">

Avoid repeating the title, stacking keywords, or reusing one description across many pages. If a platform fills it in automatically from the first lines of the page, check that those lines are useful. Cookie notices and "Free shipping this week" banners make poor summaries.

Using headings as an outline

Headings should read like a table of contents. Someone reading only your headings should understand what the page covers and where each part is.

• One h1 per page that names the page's subject, usually close to the title. The check only needs at least one, but more than one makes the page's subject less clear.
• h2s for main sections, h3s inside them. Don't skip levels or pick a heading level for its size; use CSS for size.
• Make headings descriptive. "Delivery times by region" beats "Good to know". On FAQ pages, the question itself makes a good heading.
• Use real heading tags. Bold text styled to look like a heading isn't one. Agents, and screen readers, can't see the styling.
• Keep headings in the HTML. If your page builder adds them with JavaScript, many agents won't see them. See our guide to JavaScript-only content.

A product page outline might look like this:

h1  Ethiopia Yirgacheffe whole bean coffee
  h2  Tasting notes
  h2  Brewing suggestions
  h2  Shipping and returns
  h2  Reviews

Don't forget the lang attribute

It's one attribute, and it's often missing from custom themes: <html lang="en">. Use the right code for each language version, such as lang="fr" or lang="en-GB". It helps agents interpret the page, translate it correctly and quote it in the right language. Screen readers use it to choose a voice.

Fixing it across a whole site

1. Start with templates. Titles, descriptions and headings on product, category and policy pages usually come from a theme template. Fix the pattern once and every page benefits.
2. Feed templates real data. Product name, variant, category and brand make good titles. Short product descriptions make good meta descriptions.
3. Hand-write the important ones. Your home page, top categories and policy pages deserve titles and descriptions written by a person.
4. Crawl for duplicates. Most SEO crawlers report missing and duplicate titles, descriptions and h1s across the whole site.
5. Re-run AgentScore to confirm the home page passes.

Clear labels are one part of a readable site. For the rest, see the agent readiness guide.

How agents read a price

Agents find prices in two places. The first is the text of the page: the "$18.00" a shopper sees next to the add-to-cart button. The second is structured data: a block of schema.org JSON-LD in the HTML that states the price, currency and stock as plain values. Good stores have both, and they agree.

Both have to be in the HTML your server sends. Many agents, and most AI search crawlers, read that HTML without running JavaScript. If your theme loads prices from an API after the page appears, those agents see a product with no price. To a shopper asking an assistant to compare three coffee grinders, that's a product that doesn't make the list.

What AgentScore checks

Two Readability checks cover prices.

AgentScore finds these pages from links on your home page, then from your sitemap, then a common address such as /pricing. If it can't find one, the checks are marked "not tested" rather than failed.

This guide focuses on the visible price and keeping everything consistent. For a full walk-through of product structured data, see Product data that AI shopping agents can read.

Put the price in the HTML

Open a product page, right-click and choose View Page Source (not Inspect, which shows the page after JavaScript). Search for the price. If it's there, agents can read it. If it isn't, the fix belongs to your developer or platform: render the price on the server, as part of the page, and let JavaScript update it afterwards if a shopper changes options.

<p class="price">
  <span>$18.00</span> <span>USD</span>
</p>

For software and services, the same applies to your pricing page: each plan's price, billing period and what it includes should be readable text, not an image or a widget that loads later. If you don't publish prices, say so in plain words, for example "Pricing is quoted per project. Contact sales for a quote." Then an agent can tell people how to get one, instead of reporting that it couldn't find any.

Make the currency unambiguous

"$18" means different things in the US, Canada and Australia. A person infers the currency from where they are. An agent may be working for someone elsewhere, fetching your page from a data center in another country.

• Use a currency code or prefix when it could be unclear. "US$18.00", "$18.00 USD" or "CA$24.00" leaves no doubt.
• Always set priceCurrency in structured data using the three-letter ISO 4217 code, such as USD or EUR.
• Be careful with automatic currency switching. If you change the currency based on the visitor's location, an agent may see a different currency from its user. Give each currency a stable URL or a visible switcher, and keep the currency in the page text and the data consistent.
• Say whether tax is included. "Includes VAT" or "plus sales tax" changes the answer to "how much is it?".

Sale prices and "from" prices

Discounts are where agents most often misread prices. A page showing "$24.00 $18.00" with the first one struck through looks obvious to a person, because they can see the line through it. An agent reading text may see two prices side by side.

• Label both prices in words. "Was $24.00, now $18.00" reads correctly in any form.
• Mark the old price up as removed. Use <del> or <s> for the original price, not just a CSS line-through.
• Put the price the customer will pay in structured data. The offer's price should be the current sale price. If the sale ends on a known date, priceValidUntil says so.
• Make "from" prices explicit. "From $18.00" on a product with several sizes should say what the $18.00 buys, such as "From $18.00 (340 g)".

<p class="price">
  Was <del>$24.00</del>, now <strong>$18.00 USD</strong>
  <span>Sale ends October 31</span>
</p>

Products with variants

When sizes, colors or pack sizes have different prices, agents need to know which price goes with which option. A single price that changes with JavaScript when a shopper picks a size is invisible to an agent that doesn't click.

• List variant prices in the HTML, for example in the option labels: "340 g, $18.00" and "1 kg, $44.00".
• Describe each variant in structured data. Schema.org's ProductGroup with hasVariant gives each option its own offer with its own price and availability. AgentScore reads offers inside variants too.
• Give variants their own URLs where your platform supports it, so an agent can link a shopper to exactly the option they asked about.

Keep every price consistent

An agent may see your price in several places: the page text, the JSON-LD, your product feed in Google Merchant Center or other shopping channels, and the cart. If they disagree, the agent can't know which is right. It may quote the wrong one, or treat your store as unreliable. Search engines may ignore structured data that doesn't match the page.

Unit prices help too. For groceries, coffee, cosmetics and anything sold by weight or volume, a unit price such as "$5.29 per 100 g" lets an agent compare fairly across pack sizes.

Your checklist

1. View the source of a product page and a pricing page. Is the price there?
2. Is the currency clear without knowing where the visitor is?
3. Do sale prices say which price is current, in words?
4. Does each variant's price appear in the HTML or the structured data?
5. Do the page, the JSON-LD, the feed and the cart all show the same price?
6. Run AgentScore and check both price checks pass.

Prices are one of the facts agents most need to get right before they recommend you or buy from you. For the step after that, getting the item into a basket and through checkout, see our guide to agent-ready checkout.

Why policy pages matter more now

Policy pages used to be read by a small share of careful shoppers, and by your support team. Now an assistant may read them on behalf of every shopper who asks a question about delivery or returns. Whatever it says becomes your answer, with your name on it.

That has two consequences. A clear policy page means accurate answers and confident shoppers. A muddled one means an assistant may misquote your returns window, invent a shipping rate, or decline to answer and suggest a store whose terms it could read. Neither shows up in your analytics as an error. It shows up as sales that didn't happen, or customers who expected something you never promised.

AgentScore doesn't have a dedicated check for policy pages yet, so this guide is about what you can do and test yourself.

Write so each sentence can be quoted on its own

Assistants often lift a sentence or two out of a page. Write so that any sentence still makes sense when it's quoted without its neighbors.

• Use numbers, not adjectives. "30 days", "$5.95", "3 to 5 business days". "Fast" and "generous" can't be quoted usefully.
• Name the thing in every sentence. "Returns are free" works alone. "They're free" doesn't.
• State the conditions next to the rule. If the 30 days starts from delivery, not from the order date, say so in the same sentence.
• Say what you don't do. Countries you don't ship to, items that can't be returned, services you don't offer. Assistants are often asked exactly these questions, and silence leaves them guessing.

Structure the page around real questions

Organize each page by the questions customers actually ask. Your support inbox and chat logs are the best source. Use those questions, or close versions of them, as headings, and answer directly underneath. That gives an agent a clear outline to search, and the heading itself matches the question it was asked. Our guide to titles, descriptions and headings covers how to build that outline.

A shipping page might look like this:

h1  Shipping and delivery
  h2  Where do you ship?
  h2  How much does shipping cost?
  h2  How long does delivery take?
  h2  Can I track my order?
  h2  What if my order arrives damaged?

Use plain HTML tables for anything with more than two variables, such as rates by region and speed. A table reads clearly as text; a rate chart saved as an image doesn't read at all.

Make sure agents can reach and read the page

• Put the text in the HTML. Accordions are fine if the answers are already in the page and just hidden until clicked. They're a problem if each answer is loaded by JavaScript when opened. See our guide to JavaScript-only content. AgentScore's JavaScript check looks at your home page, so check policy pages yourself with View Page Source.
• Avoid PDFs and images for policies. Some agents can read PDFs, many won't bother, and text in images is invisible to most.
• Don't hide policies inside a chat widget. A help bot can be useful for people, but agents need a page with a URL they can read.
• Link to them from every page. Footer links to Shipping, Returns, FAQ and Contact, in plain HTML, are what agents follow.
• List them in your sitemap and llms.txt. Your sitemap helps crawlers find them, and an llms.txt file can point agents straight at them with a one-line description each.
• Don't block them. Check that robots.txt and bot protection let AI assistants read help pages. See robots.txt for AI agents.

Keep one version of the truth

Contradictions are the fastest way to an inaccurate answer. If the FAQ says 30-day returns, the returns page says 14 days and a product page says "free returns for 60 days", an assistant has to pick one, and it may not pick the one you meant.

1. Choose a home for each policy. The returns page owns the returns policy. The FAQ summarizes it and links there.
2. Search your site for old terms after any policy change: old return windows, old free-shipping thresholds, discontinued services.
3. Match your structured data. If your product markup includes return or shipping details, update it with the page. See Product data that AI shopping agents can read.
4. Show a "last updated" date at the top of each policy page, so agents and people can tell how current it is.
5. Handle temporary changes explicitly. For holiday cut-off dates or a carrier delay, add a dated note at the top ("Orders placed after December 18 may arrive after December 25") and remove it when it no longer applies.

Different terms for different places or products

If your policies vary by country, membership or product type, say so clearly and early. One page per region, or a clearly labelled section for each, works better than footnotes. Name the region in the heading: "Returns from Canada", not "Other regions". If marketplace sellers on your site set their own policies, say that too, and say where the shopper can find them.

A short checklist

• Every policy has numbers, conditions and exceptions in plain sentences.
• Headings are the questions customers ask.
• The text is in the HTML, not in images, PDFs or JavaScript-loaded panels.
• Shipping, Returns, FAQ and Contact are linked from every page and listed in the sitemap.
• No two pages disagree, and each policy page shows when it was last updated.

Policy pages are where an assistant's answer turns into a sale or a lost one. Once they're clear, run AgentScore to check the rest of your site, and read the agent readiness guide for the bigger picture.

Why overlays stop AI agents

Browser agents, the kind that open a real browser and click through a site for someone, read a page through its structure and often a screenshot as well. An overlay that covers the page changes both. In the screenshot, the content is hidden or dimmed. In the page structure, a modal can make everything behind it unreachable until it's closed.

So the agent's first job is to get rid of it. That goes wrong in a few predictable ways:

• No button it can recognize. A close control drawn as an icon with no text or label has no name, so the agent can't tell it apart from any other shape on the screen.
• A fake button. A styled <div> or <span> that responds to a mouse click but isn't a real button. Many agents won't recognize it as something they can press.
• Unclear choices. "Customize", "Learn more" and "OK" side by side, or a reject option hidden behind a settings screen. The agent may pick the wrong one or stall.
• Overlays that come back. A pop-up that reappears on every page, or a second pop-up after the first is closed, costs the agent a step every time.
• Something sitting on the main button. A chat widget or sticky banner placed over "Add to cart" means the agent's click lands on the wrong thing.

For the business, this is a quiet loss. The agent doesn't complain or fill in a feedback form. It tells the person it couldn't get through, or moves on to a site where it could.

What AgentScore checks

The AgentScore check is called "Pop-ups and banners can be dismissed by agents", and it's one of the higher-weighted checks in the Navigability category. It works like this:

1. It opens your home page in a real browser at desktop size and waits for it to load.
2. It looks for anything fixed to the screen (a pop-up, a banner, a drawer) that covers 15% or more of the visible window. A slim header bar along the top doesn't count.
3. If it finds one, it reads the buttons and links inside it and looks for one with a clear dismiss label, such as "Accept", "Reject", "Decline", "Close", "Got it", "No thanks", "Continue" or a labelled ×.

A second check, "Agents can find and press your add-to-cart or sign-up button", catches the other half of the problem. It finds the main button on a product or pricing page and tests whether something else, usually a pop-up, banner or chat widget, sits on top of it. If it does, the check fails, because an agent's click would land on the overlay instead.

The usual suspects

What an agent-friendly banner looks like

Here's a cookie banner that people, screen readers and AI agents can all handle. It sits at the bottom of the page, doesn't block what's behind it, and every choice is a real button with a plain name.

<div class="consent-bar" role="region" aria-label="Cookie choices">
  <p>We use cookies to run the store and, with your OK, to measure ads.
     <a href="/pages/cookies">Cookie policy</a></p>
  <button type="button">Reject all</button>
  <button type="button">Accept all</button>
  <button type="button">Choose cookies</button>
</div>

And the most common failure, the icon-only close button, fixed in one attribute:

<!-- An agent sees: button, no name -->
<button class="modal-x"><svg>…</svg></button>
<!-- An agent sees: button "Close" -->
<button class="modal-x" aria-label="Close"><svg aria-hidden="true">…</svg></button>

A few rules that make the difference:

• Use real buttons with words. "Accept all", "Reject all", "Close", "No thanks". If the design calls for an icon, give it an aria-label.
• Keep it to the edge. A bar along the bottom leaves the page usable. A centered modal over a dimmed page forces every visitor, human or agent, to deal with it first.
• Make rejecting as easy as accepting. Several European privacy regulators have said this about cookie banners, and it also means an agent told to decline tracking can do so in one step.
• Let Escape close it where closing is allowed, and return focus to the page afterwards.
• Remember the answer. Once a choice is made, don't ask again on the next page.
• One overlay at a time. Don't stack a discount pop-up on top of a cookie banner on the first page view.

Pop-ups and the pages that earn money

The home page is where AgentScore looks for overlays, but the pages that matter most are product pages, the cart and checkout. That's where a covered button costs a sale. Check those pages on a narrower screen too: a chat launcher that sits harmlessly in a corner on a laptop can land on top of "Add to cart" on a phone-sized window.

If marketing relies on an email capture pop-up, that's a reasonable trade-off to discuss rather than a rule to break. Showing it after someone has scrolled or spent time on the page, rather than on arrival, keeps most of its value for people and removes it from the first moment an agent arrives.

How to test it

1. Open the site in a private window. That's what a first-time visitor and most agents see: no cookies, no remembered choices.
2. Try to get past every overlay with the keyboard alone. Tab to the buttons and press Enter. If you can't reach the close button, or can't tell which control has focus, an agent will struggle too.
3. Check the names. In Chrome's developer tools, the Accessibility pane shows each button's name. An empty name is a problem.
4. Visit a product page and the cart at a narrow window size and make sure nothing covers the main button.
5. Run AgentScore. It reports any overlay it found, how much of the screen it covered and which buttons it offered, and checks whether your add-to-cart or sign-up button is covered.

Overlays are one part of how agents find their way around a page. For the rest, see buttons, links and forms agents can use, and for the other thing that often greets agents on arrival, a CAPTCHA or bot challenge, see bot protection and CAPTCHAs. To see how agents handle the whole journey, not just the first screen, read how to test your key journeys with AI agents.

How agents find their way from the home page

An AI assistant answering "how much is the Pro plan at northwind.example?" or an agent asked to "buy two bags of the house blend" needs to get from your home page to the right page quickly. Agents differ in how they do that:

• Agents that read the HTML. Many AI assistants and AI search crawlers fetch the page as your server sends it and read the links in it. They don't run JavaScript, so a menu that's built in the browser is invisible to them. (See why AI agents can't see your JavaScript-only content.)
• Browser agents. These open a real browser and click around. They can see menus built with JavaScript, but they still need real links and clear names to choose from, and every extra click is a chance to get lost.

Either way, the home page is the map. If the pages that make you money aren't on it, the agent has to guess, search, or fall back on your sitemap, if it looks for one at all.

What AgentScore checks

AgentScore goes looking for your key pages the way an agent would, in this order:

1. Links in the home page's HTML. It reads every link to your own site in the page as your server sends it, before any JavaScript runs.
2. Your sitemap, if nothing suitable was linked.
3. A couple of common addresses, such as /pricing or /cart, for anything still missing.

It recognizes a key page by its address or its link text. Addresses like /products/house-blend, /pricing, /plans and /cart count, and so do links labelled "Pricing", "Plans", "Cart", "Basket" or "Bag".

The check, "Key pages are linked from the home page", then asks one question: did the home page link to them directly?

If AgentScore only found your key pages through the sitemap or a common address, the check gives a warning: the pages exist, but an agent reading your home page wouldn't have been led to them. The same pages are then used for other checks, such as whether AI agents can open them and whether prices are readable, so linking them clearly helps more than one result.

Why links go missing

Most sites that get a warning here do have links to their products and pricing. People can see them. The links just aren't where agents look.

• Menus built by JavaScript. A mega menu or mobile menu that's fetched and drawn after the page loads isn't in the HTML at all.
• Buttons pretending to be links. A "Shop now" that's a <button> or <div> with a click handler goes nowhere for an agent that reads links. So does href="#" with a script.
• Vague link text and addresses. "Discover", "Explore the range" or "See what it costs", pointing to /landing/q4-campaign, gives neither people nor agents much to go on.
• Product links only in a carousel that's loaded after the page renders, or only inside a search box.
• A cart icon with no link. A cart drawer that opens with a script, with no link to a cart page behind it.

What a good home page links to

Think of the questions people ask AI assistants about businesses like yours, and make sure the answer is one link away from the home page.

Footer links count. A plain footer with "Pricing", "Shipping" and "Returns" is often the most reliable route an agent has, because it's on every page and rarely depends on JavaScript.

What the HTML should look like

The fix is ordinary links, present in the HTML your server sends, with clear text:

<!-- Hard for agents: no link in the HTML, vague text -->
<div class="nav-item" onclick="openMenu('shop')">Discover</div>
<span class="cart-icon" onclick="openCart()"></span>
<!-- Easy for agents: real links, plain names -->
<nav aria-label="Main">
  <a href="/collections/coffee">Coffee</a>
  <a href="/collections/equipment">Brewing equipment</a>
  <a href="/pricing">Pricing</a>
  <a href="/cart" aria-label="Cart"><svg aria-hidden="true">…</svg></a>
</nav>

• Use <a href> with a real address for anything that goes to another page. A cart drawer is fine, as long as the cart icon is also a link to a cart page.
• Render the main menu on the server. If your platform or theme builds the menu in the browser, ask your developer to output at least the top-level links in the HTML.
• Name links for what's behind them. "Pricing" beats "See what it costs"; "Shop coffee" beats "Discover".
• Keep addresses readable. /pricing, /cart and /products/house-blend tell an agent what a page is before it opens it.
• Make menus open on click, not only on hover, and make the top-level items real links too, so there's always a route in.

How to check your own home page

1. View the source. In most browsers, right-click the page and choose "View page source" (not "Inspect", which shows the page after JavaScript has run). Search for href="/cart", /pricing or /products/. If they're not there, agents that read HTML can't see them.
2. Turn JavaScript off in your browser's settings and reload the home page. Whatever navigation is left is what those agents get.
3. Click through with the keyboard. Tab from the top of the page to a product and the cart. If you can't, browser agents may not either.
4. Run AgentScore. It tells you whether your products, cart or pricing were linked from the home page, or only turned up another way, such as through the sitemap.

Once agents can reach your key pages, the next questions are whether they can read them and act on them. See prices AI agents can read, buttons, links and forms agents can use and site search and filters AI agents can use.

How agents use search and filters

There are two broad ways an AI agent gets to a list of products:

• Through the page. Browser agents open your site, find the search box by its label, type, press Enter, and then click filters and sort options, much as a keyboard or screen-reader user would.
• Through the address. If your search and filtered results have their own URLs, an agent can go straight to /search?q=trail+shoes or a filtered category page. AI assistants that fetch pages rather than click through them can only work this way: they can't type into a box, but they can open a link.

A site that supports both gives every kind of agent a route to the right products. A site that supports neither forces agents to browse category pages one by one, which is slow and often ends with a worse answer.

A search box agents can find and use

The search box should be a real form that works without anything clever. That means a labelled field, a named submit button, and a results page that loads when the form is submitted.

<form role="search" action="/search" method="get">
  <label for="q">Search products</label>
  <input id="q" name="q" type="search" autocomplete="off">
  <button type="submit">Search</button>
</form>

• Label it. A visible label or an aria-label such as "Search products". A magnifying-glass icon on its own has no name an agent can read.
• Keep it visible. A search box that only appears after clicking an unlabelled icon is an extra step agents often miss. If space is tight, make the icon a real button named "Search".
• Make Enter work. Pressing Enter should submit the search and open a results page. Predictive suggestions as you type are fine, but they shouldn't be the only way to get results.
• Accept the words people use. Product names, product types, brands, SKUs, common misspellings and synonyms ("sneakers" and "trainers"). An agent passes on what the person asked for, in their words.

Give results and filters their own URLs

This is the single most useful change for AI agents, and it's good for people too: results can be bookmarked, shared and reached with the back button.

Every search and every combination of filters should be reflected in the address, using readable parameters:

https://northwind.example/search?q=trail+shoes
https://northwind.example/collections/shoes?waterproof=yes&size=10&max_price=150
https://northwind.example/collections/shoes?sort=price-asc&page=2

• Update the URL when filters change. If your filters work without reloading the page, update the address as they're applied, so the current view always has a link.
• Make the URL load the same results. Opening that link fresh, with no history, should show the same filtered list.
• Put results in the HTML. If results are only drawn by JavaScript, agents that read HTML see an empty page. See why AI agents can't see your JavaScript-only content.
• Use plain parameter names and values. size=10 and color=black are guessable. f=a8x2k isn't.

Filters as real controls

Filters are where many stores lose agents. They're often built as custom widgets that look good but aren't recognizable as checkboxes, options or buttons. The fix is to use the controls the browser already provides.

<fieldset>
  <legend>Size</legend>
  <label><input type="checkbox" name="size" value="9"> 9 (14)</label>
  <label><input type="checkbox" name="size" value="10"> 10 (22)</label>
</fieldset>
<label for="sort">Sort by</label>
<select id="sort" name="sort">
  <option value="relevance">Most relevant</option>
  <option value="price-asc">Price: low to high</option>
</select>

A few more details that save agents from guessing:

• Say how many results there are in text ("24 products"), and update it when filters change. An aria-live region lets assistive tech, and agents reading the page structure, notice the change.
• Show active filters as named buttons that remove them ("Remove filter: Size 10"), plus a "Clear all".
• Handle no results helpfully. Say plainly that nothing matched, and offer a way out: related categories, or the same search without the last filter.
• Keep filter panels reachable. On smaller screens, the "Filters" toggle should be a real, named button, not an icon.

Tell agents how your search works

Once your search has a predictable URL, you can describe it so agents don't have to discover it:

• In llms.txt. A line such as "Search: https://northwind.example/search?q={query}" in your llms.txt file hands the pattern to any AI that reads it.
• In structured data. schema.org lets a WebSite describe its search with a SearchAction. Google retired the search box it used to show for this in 2024, so don't expect a search result feature from it, but it remains a standard, machine-readable description of your search URL.
• As a tool. The most direct option is to let agents call your search as a tool, through an MCP server or WebMCP in the browser. Both are newer and still settling; see MCP and WebMCP for websites.

What AgentScore and Ghost Agents test

AgentScore doesn't have a separate check for search, but several of its checks cover the building blocks. On the home page, where most search boxes live, it checks that form fields are labelled, that buttons and links have names, and that clickable things are real buttons and links. "Content loads without JavaScript" catches a home page whose content only appears once scripts run, a sign that results and filters may be built the same way. The free scan doesn't run a shopping journey itself; a Ghost Agent test does, and a search box or filter that doesn't work shows up in its step-by-step replay.

Search is also one of the journeys Ghost Agents can test for you. You choose the journey, such as "search for a waterproof trail shoe, filter to size 10 and open the first result", and a Ghost Agent works through it like a customer would, on the schedule you set. When a theme update or a new search app breaks it, you find out before customers do. See how to test your key journeys with AI agents.

A quick checklist

1. Search box is visible, labelled, and submits with Enter to a results page.
2. Searches and filtered views have their own readable URLs that load the same results when opened fresh.
3. Results are in the page HTML, not only drawn by JavaScript.
4. Filters are checkboxes, selects, number fields and buttons with names, not styled boxes.
5. Result counts, active filters and "no results" are shown in text.
6. Pagination uses real links.
7. Your search URL is described in llms.txt, and robots.txt doesn't shut AI assistants out of it by accident.

For the controls on the rest of the page, see buttons, links and forms agents can use. For getting agents to your categories in the first place, see link your key pages from the home page.

Why forms are where agents give up

Reading a page is easy for an AI agent. Finishing a form is not. Every field has to be matched to something the person asked for, every choice has to be made, and every step has to confirm that it worked. Forms also tend to carry the things built to stop bots: CAPTCHAs, email verification and one-time codes.

None of this means removing your defenses. It means designing the form so an agent can do the routine part, and handing over to the person at the moments that genuinely need them.

The basics every form needs

• A visible label on every field. Agents match "my phone number" to a field called "Phone number". Placeholder text alone is a weak substitute. See buttons, links and forms agents can use.
• The right field types and autocomplete values. type="email", type="tel", autocomplete="given-name", postal-code and organization tell an agent exactly what each field wants.
• Only the fields you need. Every optional field is another chance to guess wrong. Ask for the rest after the lead or booking is in.
• Native controls for choices. Real <select> menus, radio buttons and checkboxes, not custom dropdowns built from <div>s.
• Errors written as text next to the field. "Enter a phone number with area code" tells the agent what to fix. A red border doesn't.
• A clear confirmation. After submitting, show a page or message that says what happened: "Thanks, Sam. Your table for 4 is booked for Friday 10 October at 7:30 pm. Reference NW-4821." The agent can read that back to the person. A toast that disappears after two seconds can't be.

Sign-up forms

For software companies, sign-up is the conversion. An agent asked to "start a trial" needs to find the sign-up page, fill it in and reach a confirmed account, or at least a clear next step for the person.

• Link to sign-up from every page. A real link with clear text ("Start free trial"), not a button that opens a modal only after a script loads. AgentScore checks that agents can find and press your sign-up button.
• Keep password rules on the page. State them before the field ("At least 12 characters"), not only in an error after submit.
• Make "Sign up with Google" an option, not the only route. Third-party sign-in opens a pop-up on another domain that many agents can't or shouldn't use. An email option keeps the door open.
• Don't hide sign-up behind a sales call. If your plans need a conversation, say so in text, and offer a booking form the agent can complete.

Email verification and one-time codes

An agent working on someone's behalf usually can't read their inbox or their text messages, and you wouldn't want it to have that access by default. So any step that says "enter the code we just sent" is a handoff point. Handle it well:

1. Let people do as much as possible before verification. Create the account, save the booking request or record the lead first, then ask for confirmation. If the agent stops at the verification step, the work isn't lost.
2. Say clearly what happens next. "We've sent a link to sam@example.com. Click it to activate your account." The agent can pass that message straight to the person.
3. Prefer a link over a short-lived code. A link the person clicks later works across devices and doesn't expire while the agent reports back. If you use codes, give them a sensible lifetime and a "resend" button with a name.
4. Mark code fields properly. autocomplete="one-time-code" and inputmode="numeric" on a single field are easier for people and agents than six separate boxes.

CAPTCHAs on forms

A CAPTCHA on every submission stops AI agents as surely as it stops spam bots. Most sites don't need one on every form. Better options:

• Risk-based challenges that only appear for suspicious submissions, such as many attempts from one address in a short time.
• Invisible checks like honeypot fields and rate limits, which stop crude bots without asking anyone to solve a puzzle.
• Recognizing verified agents. Some agents sign their requests with Web Bot Auth, and services such as Cloudflare can let verified bots through.

AgentScore checks for a CAPTCHA or challenge when an agent arrives on your site, and at the cart and checkout. See bot protection and CAPTCHAs for the full picture.

Booking forms and date pickers

Booking is where custom widgets cause the most trouble. A calendar that only works by clicking tiny cells, or a time-slot grid of unlabelled boxes, leaves an agent guessing.

Also make availability readable as text where you can, for example a list of open times on the page. An assistant answering "is there anything Friday evening?" can then answer without driving the widget at all.

<fieldset>
  <legend>Time on Friday 10 October</legend>
  <label><input type="radio" name="time" value="19:00" disabled> 7:00 pm (fully booked)</label>
  <label><input type="radio" name="time" value="19:30"> 7:30 pm</label>
  <label><input type="radio" name="time" value="20:00"> 8:00 pm</label>
</fieldset>

If your booking runs inside a third-party widget in an iframe, test it. Some booking providers build accessible widgets and some don't, and you inherit whatever they ship.

Quote and lead forms

• Split long forms into clear steps with a heading per step and a named "Next" button. Show where the agent is: "Step 2 of 3: Your project".
• Avoid free-text-only questions where a set of options would do. "Roof type" as radio buttons is easier to answer correctly than a blank box.
• Make consent explicit. Marketing opt-ins should be unticked checkboxes with clear labels. An agent should only tick them if the person asked it to.
• Give a real response time. "We'll reply within one working day" is something the agent can tell the person. "Someone will be in touch" isn't much help.

Plan the handoff to the person

Some steps belong to the person: paying a deposit, accepting terms that bind them, confirming an email address, or signing in with a password only they know. Agents generally stop and hand back at these points, and that's how it should be. Your job is to make the handoff smooth:

• Keep what the agent entered. A booking or form that survives a pause, ideally at its own URL, lets the person pick up where the agent left off.
• Put the handoff step on its own page with a clear heading, such as "Confirm your booking", so the agent knows it has done its part.
• Show a summary of everything entered before the final step, so the person can check it quickly.

How Ghost Agent Labs helps

AgentScore checks the parts of this you can see from the outside: form fields are labelled, buttons and links have names, pop-ups can be dismissed, there's no CAPTCHA on arrival, and agents can find your sign-up or add-to-cart button. Ghost Agents go further: they run the journey itself, such as sign-up or a contact form, with test data you supply, on a schedule, and show a step-by-step replay when it breaks. They don't submit payments, and the sign-up template stops short of email confirmation.

Forms are only half of task completion. For stores, the other half is checkout and guest checkout.

Why checks aren't enough

A site can pass every check for labels, named buttons and structured data and still lose agents at a custom size picker, a cookie banner that reappears on the cart page, or a checkout step that needs an account. Those failures only show up when an agent actually tries to finish the job. That's why task completion carries the most weight in agent readiness, and why it can only be measured by running real agents.

Step 1: Pick the journeys that make money

Don't test everything. Start with the two or three journeys where an agent failure costs you a customer:

Write each one the way a customer would ask an assistant: "Find a waterproof jacket in medium under $150 and add it to the cart." Specific goals give clearer results than "buy something".

Step 2: Test by hand with consumer assistants

The quickest start costs nothing. Several consumer AI assistants can now browse and act on websites, usually in an "agent" or browsing mode. Use the ones your customers are likely to use.

1. Give it the goal and your site. "On northwind.example, find a 12-cup coffee maker and add it to the cart. Don't buy anything."
2. Watch, don't help. Most agent modes show what the agent is doing. Note every hesitation, wrong click and retry, not just whether it finished.
3. Ask it what went wrong. If it fails, ask what it was trying to do and what it couldn't find. The answer often names the exact control.
4. Repeat it. Agents don't behave the same way every time. Run each journey at least three times before drawing conclusions.

Manual testing is good for discovery: it shows you how agents experience your site and builds a shared picture for the team. It doesn't scale, and it won't tell you when something breaks next Tuesday.

Step 3: Automate it with Ghost Agents

Ghost Agents are AI agents that run your key journeys for you. In the Ghost Agent Labs app, you set up a test for a site you've added:

• Goal. Start from a common journey (checkout, product search, sign-up or contact form) or describe your own in plain words.
• Start page. Where the agent begins. It can only visit pages on your site.
• When it passes. A check on the final page the agent reaches: it reached the payment step, the page shows certain text, or the address contains a certain path. These are checked directly, not by asking an AI model whether it succeeded.
• Test data. A test email address and any other values the journey needs.
• Schedule and repeats. Run it by hand, daily or hourly, with several runs per check.

Each run records every step: the page the agent saw, what it did and its reasoning. When a run fails, the replay opens on the step where it got stuck, and you can compare it with the last run that passed. Failing tests can alert your team by email or Slack.

What to measure

• Pass rate. Because agents vary, one run proves little. Run each check several times and look at the share that passed. Ghost Agents report a journey as passing, flaky (some runs passed) or failing (none did).
• Where it fails. The step where failures cluster is your fix list: the size picker, the cookie banner, the account wall.
• Steps taken. A journey that passes in 12 steps one week and 25 the next has become harder, even if it still passes.
• Change over time. A journey that passed yesterday and fails today usually means something on your site changed: a theme update, a new app, a new pop-up.

Keep testing safe

Testing with agents means letting software click buttons on your live site. Set some rules:

• No real purchases. Stop at the payment step. Ghost Agents stop when they reach a payment form, never enter card details, and refuse buttons that place orders, pay, or delete or cancel accounts. When testing by hand, tell the assistant not to buy, and stay in control at payment.
• Test data only. Use a dedicated test email address, such as one on a domain you control, and dummy names and phone numbers. Never use real card or bank details, or a real customer's details.
• Watch your side effects. Test sign-ups create accounts, test leads land in your CRM and test bookings take slots. Use recognizable test values (a "test+" email or the name "Ghost Test") so your team can filter them out, and avoid submitting booking forms that hold real inventory.
• Tell the people who need to know. Let whoever runs bot protection and analytics know tests are coming. Ghost Agents identify themselves with GhostAgent/1.0 in the user agent and sign their requests with Web Bot Auth, so they can be recognized and filtered from reports.
• Consider staging for risky journeys. If a journey can't be tested on the live site without real consequences, test it on a staging copy that matches production.

Turn results into fixes

A failed run is only useful if someone acts on it. For each failure, record the journey, the step, what the agent was trying to do and a screenshot, then route it to whoever owns that part of the site: the developer for an unlabelled button, the platform admin for forced account creation, marketing for the pop-up. The replay usually tells you which.

Most fixes are the same ones covered elsewhere in these guides: buttons and forms agents can use, cookie banners and pop-ups, checkout and sign-up and booking forms.

Where to start this week

1. Run AgentScore on your home page to clear the obvious blockers first.
2. Pick your single most valuable journey and try it by hand with an AI assistant, three times.
3. Set it up as a Ghost Agent test with a daily schedule and a pass condition on the final page, so you hear about it the day it breaks.

Two different things to measure

AI shows up in your data in two ways, and it helps to keep them apart:

• AI agents visiting your site. Crawlers and assistants that read your pages, mostly without running your analytics tag. These only show up in server or CDN logs. See how to measure AI agent traffic.
• People arriving from AI assistants. A person reads an AI answer, clicks a link to your site, and browses in their own browser. Your analytics tag runs normally. This is AI referral traffic, and it's what this guide covers.

The first tells you whether AI can read and use your site. The second tells you whether that's turning into visits and sales.

How AI referrals show up

Analytics tools work out where a visit came from in two main ways: the referrer, which the browser sends to say which site the person came from, and campaign tags (UTM parameters) on the link itself.

Referrers

When someone clicks a link in a web-based assistant, the browser often sends that assistant's domain as the referrer. Domains you're likely to see include:

Treat this list as a starting point rather than a complete map. Assistants change domains, launch new apps and change how they link out, so check your own referral report for new sources every month or so.

Campaign tags

Some assistants add their own tag to outbound links. ChatGPT, for example, has been adding utm_source=chatgpt.com to many links in its answers. In your analytics that shows as the source chatgpt.com, sometimes with no medium, or alongside the referrer. Other assistants may do the same, or nothing at all.

Visits that arrive with no label

Not every AI referral announces itself. Mobile and desktop apps often open links without sending a referrer, privacy settings and browsers can strip it, and a person who copies a URL from an answer and pastes it into a new tab arrives as "direct". Some of your direct traffic is AI referral traffic you can't see. Treat any AI referral number as a floor, not a total.

Reporting on it in Google Analytics 4

Analytics tools have been changing how they group this traffic. In GA4's default channel grouping, AI assistants have typically ended up under Referral, mixed in with every other website that links to you. Check what your property shows today, because default channel definitions get updated. If AI referrals aren't already separated, two approaches work:

1. A quick view. In the Traffic acquisition report, change the dimension to "Session source" and filter for the domains above. Good for a one-off look.
2. A custom channel group. In Admin, under Data display, create a channel group with an "AI assistants" channel whose rule matches the session source against a pattern, and place it above Referral so it wins. Custom channel groups apply to your reports going back in time, so you see history too.

A source pattern to start from:

(^|\.)(chatgpt\.com|chat\.openai\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai)$

Use a "matches regex" condition on session source. Keep the pattern in one place, such as a shared doc, and add new assistants as they appear in your referral report. Other analytics tools work the same way: a rule on referrer or source that puts these visits in their own channel.

What to measure

• Sessions and trend. AI referral sessions by assistant, week over week. The direction matters more than the size today.
• Landing pages. Which pages assistants send people to. Often it's product pages, comparison content or policy pages rather than the home page. Those pages now act as front doors, so make sure they convert.
• Engagement and conversion. Conversion rate, revenue and leads from the AI channel compared with search and other referrals. People arriving from an AI answer have often already been given a recommendation, so watch how their behavior compares.
• Assisted conversions. Someone may first hear about you from an assistant and buy a week later through a branded search. In GA4, compare attribution models or look at conversion paths to see where AI sources appear early in the journey.

The limits of attribution

Be honest with stakeholders about what this data can and can't show:

• It undercounts. App traffic and copied links arrive as direct. The real number is higher than what you see.
• It only counts clicks. Many AI answers mention a brand without a click. Someone who reads "Northwind has the best return policy" and later types your address directly is AI-influenced but won't appear here.
• It doesn't count agent purchases. When an assistant buys through an agentic commerce protocol, there may be no web session at all. Those orders show up in your commerce platform, so ask your platform how they're tagged.
• Browser agents blur the line. An agent browsing on someone's behalf may run your tag and look like a normal visitor. Some of what looks like human traffic may be agents.

Getting more AI referral traffic

Assistants recommend and link to sites they can read and trust. The groundwork is the same as for agent readiness: let AI search and assistant agents in through robots.txt, put key content in the HTML, describe products and prices with structured data, and make policy pages easy to quote. If you block OAI-SearchBot or PerplexityBot, don't expect referrals from ChatGPT search or Perplexity.

Then make sure the visits convert. The person arriving from an AI answer, or the agent acting for them, needs a page that works. AgentScore shows how your site looks to AI agents, and Ghost Agents test the journeys that turn a referral into a sale.

Why it matters now

AI assistants increasingly visit websites on a person's behalf: to answer a question, compare products, check a policy, or buy something. Each of those visits stands in for a customer. If the agent can't use your site, it doesn't complain or call support. It moves on to a site it can use, and recommends that one instead.

Agent readiness is not the same as SEO, though they overlap. SEO is about being found and ranked. Agent readiness is about what happens next: whether an agent that has found you can actually do something useful with your site.

Not every agent wants the same thing. A crawler collecting pages for AI training is very different from an assistant checking your returns policy for a customer, or a browser agent trying to check out. See the AI agents that visit your website, and keep the agent readiness glossary handy for the terms in this guide.

The four parts of agent readiness

AgentScore measures agent readiness out of 100, across four categories. The weights reflect what stops agents most often in practice.

1. Access: can agents get in?

Before anything else, the agent has to be allowed through the door. Most access failures are accidental: a robots.txt rule copied from a template, or bot protection that challenges every visitor that isn't a person.

• robots.txt allows AI assistants and AI search agents. You can still block training crawlers separately. See robots.txt for AI agents.
• Bot protection doesn't block AI agents that a person would get through, and can tell real agents from impostors. See bot protection and CAPTCHAs and how to tell if an AI crawler is real.
• No CAPTCHA or challenge page on arrival for ordinary pages.
• An llms.txt file points agents to your most important pages. See how to write an llms.txt file.

2. Readability: can agents understand your pages?

Many agents read the HTML your server sends and never run JavaScript. If your prices, product details or policies only appear after scripts load, those agents see an empty page.

• Key content is in the HTML, through server-side rendering or static generation, not only added by JavaScript. See why AI agents can't see JavaScript-only content.
• Structured data (schema.org) describes products, prices, availability and your business. See product data AI shopping agents can read.
• Clear titles, headings and image descriptions on every important page.
• A valid sitemap that lists the pages you want found.

3. Navigability: can agents find their way around?

Browser agents find their way around a page much as a screen reader does: by the names and roles of buttons, links and form fields. Much of agent readiness here is simply good accessibility. See buttons, links and forms agents can use.

• Every button, link and form field has a name an agent can read. An icon with no label is invisible.
• Cookie banners and pop-ups can be dismissed and don't cover the content.
• Menus open on click or focus, not only on hover.
• Page landmarks such as header, navigation, main and footer let agents find what they need quickly.

4. Task completion: can an agent get the job done?

This is the part that matters most, and the only way to measure it is to try. A site can pass every technical check and still lose agents at the size selector or the checkout button.

• Online stores: an agent can search, find a product, choose options and add it to the cart. See how to make checkout work for AI shopping agents.
• Software companies: an agent can find pricing and reach the sign-up form.
• Service businesses: an agent can find prices, availability and a way to book.

The free AgentScore scan doesn't test this yet: it marks Task completion “not tested” and works out your score from the other three categories. Ghost Agents test it today, running the journeys you choose on a schedule, with a step-by-step replay of each run, and alerting you when one breaks.

How to measure your agent readiness

1. Run AgentScore. It's free, takes under a minute, and checks Access, Readability and Navigability. Task completion isn't part of the free scan yet.
2. Look at your agent traffic. Your analytics probably can't see agents, because most never run the JavaScript tag. Server or CDN logs can, and Ghost Agent Labs reads them for you. See how to measure agent traffic.
3. Test your money journeys. Checkout, sign-up and booking are the journeys where an agent failure costs you a customer.

A 30-minute starter checklist

Common questions

Won't letting agents in mean letting scrapers in?

No. Agent readiness is about letting in the agents you want, and the main ones can be verified. Keep blocking unknown and abusive bots; just make sure your rules don't catch AI assistants that are bringing you customers.

Do I have to let AI companies train on my content?

No. Training crawlers and the agents that fetch pages for people use different names, so you can block one and allow the other. Our robots.txt guide shows how.

Is this just accessibility?

There's a big overlap, especially in navigation, and work on one helps the other. But agent readiness also covers access rules, machine-readable data, and whether an agent can finish a real task.

Three kinds of AI agent

"AI bots" isn't one thing. The major AI companies each run several agents with different jobs and different names, and the right policy for each is different.

Some names aren't crawlers at all. Google-Extended and Applebot-Extended are control tokens: Google and Apple crawl with Googlebot and Applebot as usual, and these names let you say whether that content may be used for their AI models. Blocking them doesn't affect normal search.

Lists like this change as companies launch new agents, so check each operator's documentation for the current names.

A sensible default

For most businesses, the agents that matter most are AI search and assistant fetchers, because they answer people's questions about you and link back to you. Training is a separate decision about your content, and it's yours to make.

This robots.txt allows everything, which is also what you get with no robots.txt at all:

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

If you'd rather not have your content used to train AI models, but still want to appear in AI search and assistant answers, block only the training names:

# Allow AI search and assistants; opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
Disallow: /
User-agent: *
Disallow: /cart
Disallow: /account
Allow: /
Sitemap: https://www.example.com/sitemap.xml

Listing several User-agent lines above one set of rules applies those rules to all of them.

The rule that catches most people out

A crawler follows only the most specific group that matches its name, and ignores the rest. If there's a group for GPTBot, GPTBot ignores everything under User-agent: *.

So in the example above, the training crawlers don't inherit the /cart and /account rules; they don't need them, because they're blocked from everything. But the reverse mistake is common:

User-agent: *
Disallow: /checkout
Disallow: /admin
User-agent: OAI-SearchBot
Allow: /

Someone added that second group to "make sure" OpenAI's search crawler gets in. Instead, they've told it that /checkout and /admin are open, because it no longer reads the * group. If you add a group for a specific agent, copy into it every rule you want it to follow.

Other common mistakes

• A leftover Disallow: /. Staging sites are often launched with a robots.txt that blocks everything. Check yours today.
• Blocking every AI name from a copied list. Long "block all AI" lists usually include the assistant fetchers and AI search crawlers too, which removes you from AI answers as well as training.
• Blocking the pages agents need. Disallowing /products or /pricing for all bots, or for *, to save crawl budget also hides them from agents.
• Relying on robots.txt to block bad bots. robots.txt is a request, not a lock. Well-behaved crawlers follow it; scrapers ignore it. Abusive traffic has to be stopped at your CDN or firewall.
• Forgetting your CDN. A firewall rule or bot setting that challenges "AI crawlers" overrides a friendly robots.txt. The agent never gets far enough to read it.

A note on assistant fetchers

When a person asks an assistant to read a specific page, some operators treat that fetch like a person clicking a link rather than a crawl, and say robots.txt rules may not apply to it. If you need to stop those visits completely, that has to happen at your CDN or server, not in robots.txt. For most businesses, though, these are the visits you want most: each one is a customer asking about you.

How to check your robots.txt

1. Open https://yourdomain.com/robots.txt and read it top to bottom. Note every group and which names it applies to.
2. For each AI search crawler and assistant fetcher in the table above, work out which group it follows, remembering the most-specific-group rule, and whether that group blocks your important pages.
3. Run AgentScore. Its robots.txt check reads your rules as 21 AI agents would and tells you which assistants and AI search agents are blocked from your home page. Blocked training crawlers are reported but don't cost you points, because that's a legitimate choice.

robots.txt is only the first door. Next, make sure your bot protection lets the real agents through, and can tell them apart from impostors using their names. See how to tell if an AI crawler is real.

Why user agents can't be trusted

Every request names its sender in the User-Agent header, and that name is a promise, not a proof. Scrapers routinely borrow the names of well-known crawlers, because many sites let those crawlers through without a challenge. If your bot rules trust the name alone, you're letting in anyone who copies it. If they distrust it, you're blocking real agents along with the fakes.

Verification resolves that. It answers one question: did this request really come from the company it names?

Method 1: published IP ranges

Several operators publish the IP addresses their crawlers use, as machine-readable JSON files. If a request claims to be one of their agents and comes from an address in the list, it's real.

Each operator links its list from its crawler documentation. A few practical points:

• Refresh the lists often. Ranges change. Fetch them at least daily, and don't treat a request as fake because it's missing from a list that's days old.
• Match the list to the agent. An OpenAI address in the GPTBot list doesn't verify a request claiming to be ChatGPT-User.
• Handle IPv6. Several lists include IPv6 ranges.

Method 2: forward-confirmed reverse DNS

Some operators instead promise that their crawlers' IP addresses resolve to a host name on their own domain. The check has two steps, because reverse DNS on its own can be faked by whoever controls the IP address:

1. Reverse lookup: look up the host name for the request's IP address, and check it ends in the operator's domain.
2. Forward lookup: look up that host name's IP addresses, and check the original IP is among them.

Google's own example, from a terminal:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The name ends in googlebot.com and points back to the same address, so the request is from Google. Domains that operators document for this include:

DNS lookups are slow compared with a page request, so cache the result for each IP address rather than checking on every hit.

Method 3: signed requests (Web Bot Auth)

The newest method doesn't depend on IP addresses at all. With Web Bot Auth, the agent signs each request with a private key and publishes the matching public key on its own domain. Your CDN or server checks the signature, and a valid one proves the sender, whatever network it came from.

That makes it the best fit for browser agents running in the cloud, whose addresses change constantly. It's still early, so check whether your CDN supports it. We explain how it works, and why we sign all of our own agents' requests, in Why we sign every request our agents send.

When there's no proof at all

Not every operator publishes IP ranges, a reverse DNS domain, or signing keys for every agent. For those, you can't prove a request is real or fake; you can only weigh the evidence, such as how it behaves and how fast it requests pages.

Treat "can't tell" as its own answer, not as "fake". Blocking everything you can't verify will block legitimate agents from operators that simply haven't published proof yet.

Putting it together

1. Identify: match the user agent to a known agent and its operator.
2. Verify: use the strongest proof that operator offers: a signature, then published IP ranges, then reverse DNS.
3. Decide: let verified agents you want through; challenge or block requests that claim a name but fail verification; and handle "can't tell" with ordinary rate limits rather than a hard block.

Why good agents get caught

Bot protection, whether it's a CDN feature, a web application firewall (WAF) or a dedicated product, looks for signs that a visitor isn't a person. AI agents show many of those signs, for entirely legitimate reasons:

• They say they're bots. Honest agents announce themselves in their user agent, and a rule that blocks "bots" or "AI crawlers" by name catches them first.
• They don't run JavaScript. Many challenges work by running a script in the visitor's browser. Assistant fetchers and AI crawlers usually read only the HTML, so they never pass.
• They come from data centers. Agents run in the cloud, and cloud IP addresses score as higher risk than home broadband.
• Browser agents look automated. Even agents that drive a real browser move and type differently from people, and fingerprinting tools notice.
• One-click "block AI" settings. Several providers offer a single switch to block AI bots. Depending on how it's set up, it can block assistants and AI search along with training crawlers.

Signs it's happening to you

• AI assistants say they "can't access" your site, or describe it from out-of-date information.
• Your CDN or firewall logs show 403 (forbidden), 429 (too many requests) or challenge responses for agents like ChatGPT-User, Claude-User or Perplexity-User.
• You appear in AI search results much less often than competitors with weaker SEO.

The quickest test is AgentScore. It requests your home page as a normal browser and as five AI agents, including ChatGPT-User, Claude-User, Perplexity-User and OAI-SearchBot, and compares what comes back. It fails the check if an agent is blocked or challenged while the browser gets through, and warns if an agent gets less than half the content the browser does. Separate checks look for CAPTCHAs on your home page, and test whether agents can open your product, pricing and cart pages.

How to fix it

1. Allow verified agents, not names

The safe way to let agents in is by verified identity. Most bot management products keep a list of verified bots, checked against the operator's published IP ranges, reverse DNS or signatures, and let you allow categories of them. Allow the categories for AI assistants and AI search, and keep blocking requests that only claim those names. Our guide to telling real AI crawlers from fakes explains how verification works.

2. Separate training from assistants

If you've turned on a "block AI bots" setting, check exactly which agents it covers. If your goal is to opt out of AI training, block only training crawlers, and leave assistant fetchers and AI search crawlers allowed. See robots.txt for AI agents for which names are which.

3. Protect actions, not pages

Abuse mostly targets actions: logging in, creating accounts, applying discount codes, checking gift card balances, submitting payment. It rarely needs protecting against on pages that only display information. Concentrate strict rules and challenges on:

• Login, sign-up and password reset forms
• Checkout submission and payment
• Gift card, coupon and stock-check endpoints that attackers hammer
• Search and APIs, with rate limits rather than outright blocks

Keep product, category, pricing, policy and help pages as open as you safely can. Those are the pages agents need to answer questions about you.

4. Use rate limits instead of blocks

A real assistant fetcher makes a handful of requests because a person asked about you. A scraper makes thousands. A sensible rate limit per IP address or verified agent stops the scraper without punishing the assistant, where a blanket block stops both.

5. Replace puzzle CAPTCHAs on key pages

AI agents can't solve CAPTCHAs, and aren't meant to. If you need a challenge, use invisible or risk-based challenges that only escalate when traffic looks abusive, and keep them off the pages agents need to read. Never put a CAPTCHA in front of ordinary content on arrival.

6. Don't serve agents a different page

Some setups don't block agents outright, but quietly serve them a stripped-down page, a placeholder, or an "enable JavaScript" message. To the agent this is as bad as a block, and it's harder to notice. Agents should get the same content a browser does.

A quick checklist

To keep an eye on this over time, watch how often agents are blocked in your traffic. See how to measure AI agent traffic.

What structured data is

Structured data is a small block of machine-readable information in your page's HTML that describes what the page is about, using the shared vocabulary at schema.org. The most common format is JSON-LD: a <script type="application/ld+json"> tag that people never see, but search engines and agents read directly.

Search engines have used it for years to show prices and star ratings in results. For AI agents it matters even more. A person can glance at a page and see that "$24" next to the "Add to cart" button is the price. An agent reading the raw page has to infer that, and on a busy page with sale prices, bundle prices and "from" prices, it can get it wrong. Structured data removes the guesswork.

A complete product example

Here's JSON-LD for a single product page, with the details agents use most:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Ethiopia Yirgacheffe Whole Bean Coffee, 340 g",
  "description": "Light roast single-origin coffee with notes of jasmine and lemon.",
  "sku": "NW-ETH-340",
  "gtin13": "0123456789012",
  "brand": { "@type": "Brand", "name": "Northwind Coffee" },
  "image": "https://northwind.example/images/ethiopia-340.jpg",
  "url": "https://northwind.example/products/ethiopia-yirgacheffe",
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.7",
    "reviewCount": "212"
  },
  "offers": {
    "@type": "Offer",
    "price": "18.00",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock",
    "itemCondition": "https://schema.org/NewCondition",
    "url": "https://northwind.example/products/ethiopia-yirgacheffe",
    "hasMerchantReturnPolicy": {
      "@type": "MerchantReturnPolicy",
      "applicableCountry": "US",
      "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
      "merchantReturnDays": 30
    }
  }
}
</script>

The fields that matter most

Products with options

If a product comes in sizes or colors with different prices or stock, describe each variant. Schema.org's ProductGroup type groups variants under one parent, with variesBy naming what changes (such as size or color) and hasVariant listing each Product with its own offer. At minimum, make sure the price and availability on the page match the variant a visitor lands on.

Beyond product pages

• Home page: Organization (or LocalBusiness with address and opening hours) with your name, logo, website, contact details and social profiles. AgentScore looks for structured data on your home page.
• Category pages: BreadcrumbList shows where a page sits in your catalog, and helps agents move between categories.
• Help pages: FAQPage for genuine question-and-answer content, such as shipping and returns questions.
• Software pricing: SoftwareApplication or Product with an Offer for each plan.

Rules that keep it trustworthy

• Match the page. Every value must match what a person sees. A price in the markup that differs from the page is worse than none: an agent may quote the wrong one, and search engines may ignore your markup.
• Keep it current. Generate it from the same data as the page, so prices and stock update together. Hand-written markup goes stale.
• Put it in the HTML. Markup added by JavaScript after the page loads is invisible to agents that don't run JavaScript. AgentScore flags this as a warning. See why AI agents can't see JavaScript-only content.
• Prefer JSON-LD. Microdata and RDFa work, but JSON-LD keeps the data in one tidy block that's easier to read and maintain.
• Don't mark up what isn't there. No ratings without visible reviews, no FAQ markup for questions that aren't on the page.

How to add it

1. Check what you already have. Many e-commerce platforms and themes output basic product markup. View the page source and search for application/ld+json.
2. Fill the gaps. Common missing pieces are availability, identifiers like GTIN, and return and shipping details. Platform apps and SEO plugins can add these, or your developer can extend the theme template.
3. Validate. Paste the page URL into the Schema Markup Validator to check the markup is valid, and Google's Rich Results Test to see which fields Google can use.
4. Run AgentScore. The structured data check passes when your home page has JSON-LD in its HTML, and warns if it only appears after JavaScript runs.

Structured data is one part of being readable to agents. The other big one is making sure your content is in the HTML at all; see the guide to JavaScript-only content, or the full agent readiness guide.

How agents read a page

When a browser loads a page, it does two things: it downloads the HTML your server sends, then runs the JavaScript, which may fetch more data and build much of what you see. Agents differ in how far they go:

Googlebot does run JavaScript, which is why many sites that rely on it rank fine in Google search. That has hidden the problem: the same site can be well indexed by Google and nearly blank to AI assistants.

How to check your own site

1. View the source. In your browser, use View Page Source (not the developer tools inspector, which shows the page after JavaScript). Search for a product name or price you can see on the page. If it isn't in the source, it's added by JavaScript.
2. Turn JavaScript off. In Chrome's developer tools, open the command menu, type "Disable JavaScript", and reload. What's left is roughly what an HTML-only agent sees.
3. Fetch it from the command line. curl -s https://yourdomain.com/your-page | grep -i "your price" shows exactly what your server sends.
4. Run AgentScore. It compares the text in your HTML with the text after JavaScript runs. It passes when at least 70% of the text is already in the HTML, and warns or fails below that.

What usually goes missing

• Prices and stock loaded from an API after the page appears, so the HTML says "Loading…" or nothing at all.
• Whole pages in single-page apps, where the HTML is an empty <div id="root"> and everything else is built in the browser.
• Product lists with infinite scroll that load more items only as a person scrolls, with no paginated links to follow.
• Content inside tabs and accordions fetched only when clicked, such as specifications, sizing and shipping details.
• Structured data injected by a tag manager or script, which agents reading the HTML never see. See product data AI shopping agents can read.
• Reviews from third-party widgets that load in the browser.

How to fix it

The aim is simple: the important content should be in the HTML your server sends. JavaScript can still make the page interactive afterwards.

Server-side rendering (SSR)

The server builds the full HTML for each request, then JavaScript takes over in the browser. Modern frameworks support this directly: Next.js and Remix for React, Nuxt for Vue, SvelteKit for Svelte, and Angular's server rendering. If your site is a single-page app built with one of these, turning on server rendering for key pages is usually the biggest single fix.

Static generation

Pages are built to HTML ahead of time, at deploy. It's ideal for content that changes rarely, such as marketing pages, docs, policies and blog posts. Astro, Eleventy, Hugo and the frameworks above can all do it, often alongside server rendering for pages that change often.

Render the essentials, enhance the rest

You don't need to render everything on the server. Prioritize what agents ask about: product name, price, availability, key specifications, shipping and returns, and plan pricing. Personalized recommendations, recently viewed items and chat widgets can stay in JavaScript.

Pre-rendering as a stopgap

Pre-rendering services run your JavaScript ahead of time and serve the finished HTML to bots. It can work as a short-term fix, but it means maintaining two versions of each page, and it depends on correctly recognizing every agent. Google describes serving bots differently ("dynamic rendering") as a workaround rather than a long-term solution. Server rendering is better.

Smaller fixes that help

• Replace infinite scroll with, or add, real paginated links (?page=2) agents can follow.
• Put tab and accordion content in the HTML and hide it with CSS, instead of fetching it on click.
• Move JSON-LD from your tag manager into the page template.
• Include a summary of reviews (average rating and count) in the HTML, even if the full reviews load later.
• Give the page a meaningful <title> and <h1> in the HTML, never only after JavaScript runs.

Once your content is in the HTML, the next question is whether agents can find their way around it and use it. See buttons, links and forms agents can use.

How agents find their way around

Most browser agents work from the page's accessibility tree, the same structured view of the page that screen readers use, often alongside a screenshot. In that tree, every element has a role (button, link, text box, heading) and a name ("Add to cart", "Search", "Email address"). The agent decides what to do by reading those names.

That's good news: almost everything that makes a site work for agents also makes it work for people using screen readers and keyboards. If you've invested in accessibility, you're most of the way there.

1. Give every button and link a name

The most common failure is the icon-only control: a cart, search, menu or close button that shows an icon and has no text.

<!-- An agent sees: button, no name -->
<button><svg>…</svg></button>
<!-- An agent sees: button "Cart, 2 items" -->
<button aria-label="Cart, 2 items"><svg aria-hidden="true">…</svg></button>

• Use visible text where you can. Use aria-label where the design calls for an icon alone.
• For a linked image, the image's alt text becomes the link's name.
• Make names specific. Ten links called "Learn more" or "Shop now" tell an agent nothing about where each goes; "Shop running shoes" does.

2. Use real buttons and links

A <div> with a click handler looks like a button to a person but isn't one to an agent: it has no role, can't be focused, and often isn't recognized as clickable at all.

<!-- Looks clickable, isn't recognized as clickable -->
<div class="btn" onclick="addToCart()">Add to cart</div>
<!-- Recognized as a button by every agent -->
<button type="button" onclick="addToCart()">Add to cart</button>

• Use <a href> for anything that goes to another page, and <button> for anything that does something on this page.
• If you truly can't change the element, add role="button" and tabindex="0", and handle the Enter and Space keys.
• Make links real URLs. A link with href="#" and a script gives an agent nowhere to go if the script doesn't run.

3. Label every form field

Agents fill in forms by matching what they've been asked to enter ("my email", "ship to 10001") to the field's label. Placeholder text is a weak substitute: it disappears once someone types, and not every agent reads it.

<label for="email">Email address</label>
<input id="email" name="email" type="email" autocomplete="email" required>

• Connect each <label> to its field with for and id, or wrap the field in the label.
• Use the right type (email, tel, number) and autocomplete values (given-name, postal-code, address-line1). They tell an agent exactly what each field wants.
• Prefer native <select>, checkboxes and radio buttons over custom-built dropdowns. Custom size and color pickers are a frequent place agents get stuck.
• Show errors as text next to the field and link them with aria-describedby, so an agent knows what to fix. A red border alone isn't enough.

4. Get pop-ups and banners out of the way

Cookie banners, newsletter pop-ups and region selectors that cover the page on arrival are one of the biggest blockers. An agent has to work out how to dismiss them before it can do anything else, and if it can't, it's stuck.

• Close and accept controls must be real buttons with clear names: "Accept all", "Reject all", "Close".
• Keep consent banners small, at the edge of the screen, rather than covering the content.
• Don't show sign-up pop-ups on arrival. If you must, wait until someone has engaged with the page.

5. Make menus work without hover

Agents click; they don't hover. A menu that only opens when the mouse rests over it may never open for an agent, hiding every category inside it.

• Make the top-level menu item a button that opens the menu on click, with aria-expanded showing whether it's open.
• Make sure the top-level items are also real links to category pages, so there's always a route in.

6. Mark up the page's landmarks

Landmarks tell an agent where the main content is, and where the navigation is, so it can skip straight to what matters.

<header>…</header>
<nav aria-label="Main">…</nav>
<main>…</main>
<footer>…</footer>

Use one <h1> per page and headings in order, so the page has an outline an agent can follow.

How to test it

1. Tab through the page. Use only the keyboard to reach the menu, search, a product, its options and the cart. Anywhere you get stuck, an agent probably does too.
2. Look at the accessibility tree. In Chrome's developer tools, the Accessibility pane shows each element's role and name. Look for buttons with no name.
3. Run an accessibility checker, such as Lighthouse in Chrome or axe, for a quick list of unnamed controls and unlabelled fields.
4. Run AgentScore. It renders your home page in a browser and checks named buttons and links, real controls, form labels, landmarks and pop-ups that cover the page. To see where a real AI agent gets stuck on a journey, run a Ghost Agent test.

For the bigger picture, see What is agent readiness?

How agents shop today

Agents buy in two ways, and you'll want to support both:

• Through your website. A browser agent uses your site the way a person would, and typically hands control back to the person to sign in or pay. This works today on any site, as long as the agent can get through.
• Through an agentic commerce protocol. The assistant completes the purchase through a direct integration, without driving your web pages. Examples include the Agentic Commerce Protocol used by ChatGPT and Google's Universal Commerce Protocol. Support usually comes through your commerce platform or payment provider.

Protocols are growing fast, but for now most agents still shop through the website. That's the journey this guide covers.

Step 1: Find the product

• Search works without JavaScript tricks. A real search form with a labelled field and a submit button, whose results have their own URL (like /search?q=decaf), lets agents search directly.
• Categories are real links. Category and product links should be ordinary <a href> links in the HTML, reachable without hover menus. See buttons, links and forms agents can use.
• Product pages describe the product. Name, price, stock and key specifications in the HTML, with structured data. See product data agents can read.

Step 2: Choose options

This is where agents most often fail on otherwise good sites. Size, color and quantity pickers are frequently custom-built swatches that look fine to people but have no names or roles agents can read.

• Use native <select> elements or radio buttons, or give custom swatches a role and a name, for example a radio group with each option labelled "Size 10" or "Color: Navy".
• Show unavailable options as disabled, with text saying why ("Out of stock"), not just a faded color.
• Update price and stock text on the page when an option changes, so the agent can confirm it picked correctly.
• Give each variant its own URL where possible (?variant=…), so an agent can link straight to the exact item.

Step 3: Add to cart

• The add-to-cart control is a real <button> with clear text, and nothing covers it, such as a cookie banner or chat widget.
• Confirm the add in text: "Added to cart: Ethiopia Yirgacheffe, 340 g" with a link to the cart. An animation alone leaves the agent unsure whether it worked.
• If a cart drawer slides out, make sure it has a labelled close button and a real link to the cart page.
• Skip upsell pop-ups at this step, or make them easy to dismiss with a labelled button.

Step 4: The cart

• The cart lives at a stable URL, such as /cart, that agents can open directly.
• Line items, quantities, subtotal and any discounts are shown as text.
• Quantity controls and remove buttons have names ("Remove Ethiopia Yirgacheffe"), not just "×" or a trash icon.
• The checkout button is a clearly named button or link, visible without scrolling past recommendations.

Step 5: Checkout

• Offer guest checkout. Forcing account creation is a dead end for many agents. Offer sign-in as an option, not a gate.
• Label every field and use autocomplete. autocomplete="shipping street-address", postal-code, email and tel tell an agent exactly where each detail goes.
• Make shipping options readable. Each option's name, price and delivery estimate as text, with a real radio button to choose it.
• Write errors as text. "Enter a 5-digit ZIP code", next to the field, rather than a red outline.
• Show the full total before payment, including shipping and tax, so the agent can confirm it with the person before they pay.
• Avoid a CAPTCHA on every checkout. Use risk-based checks that only challenge suspicious orders. See bot protection and CAPTCHAs.

Step 6: Payment

Most agents stop at payment and hand back to the person, and that's how it should be. Make the handoff smooth: a clear payment step, wallet options like Apple Pay, Google Pay or Shop Pay that a person can approve in one tap, and a cart that survives the handoff instead of emptying after a short timeout.

To let assistants complete purchases on their own, ask your commerce platform or payment provider which agentic commerce protocols it supports. AgentScore checks whether your site advertises one.

How to test your checkout

1. Walk it with the keyboard only. From the home page to the payment step, without a mouse. Anywhere you get stuck, an agent probably does too.
2. Run AgentScore. For online stores, it checks that AI agents can open your product, cart and checkout pages, find and press the add-to-cart button, and fill in labelled checkout fields, and that there's a guest checkout. It doesn't run a full journey itself.
3. Monitor it with Ghost Agents. They run the whole journey on a schedule with test details you provide, and alert you when it breaks. They stop when they reach a payment form, never enter card details, and refuse buttons that place orders, so testing never creates a real order.

Why analytics misses agents

Tools like Google Analytics count a visit when a small JavaScript tag runs in the visitor's browser and reports back. That works for people. It fails for AI agents for three reasons:

• Most agents never run the tag. AI crawlers and assistant fetchers download the HTML and leave without running JavaScript, so the tag never fires.
• Known bots are filtered out on purpose. Analytics tools exclude known bot traffic so it doesn't distort your human numbers. That's right for marketing reports, but it hides exactly the visits you want to see.
• Browser agents blend in. Agents that do run JavaScript often look like ordinary browsers, so they're counted as people, mixed in with everyone else.

You may see AI in your analytics as referral traffic: people who click a link in a ChatGPT or Perplexity answer and arrive on your site. That's valuable, but it's the result of agent visits, not the visits themselves. It doesn't show which agents read your pages, which they couldn't, or what they saw.

Where agent visits do show up

Every request to your site, from a person or an agent, passes through your web server or CDN, and they can log it whether or not any JavaScript runs. Each log line typically includes:

Getting your logs

• Cloudflare: Logpush sends HTTP request logs to storage or another service. Availability depends on your plan.
• Vercel: log drains stream request logs to an endpoint you choose.
• Fastly, Akamai and Amazon CloudFront all offer access logs or real-time log streaming.
• Your own servers: Nginx and Apache write access logs by default.
• Hosted platforms such as Shopify don't usually give you raw request logs, which makes agent traffic much harder to see. Ask your platform what bot and crawler reporting it offers.

A quick look with the command line

If you have an Nginx or Apache access log in the standard "combined" format, this counts requests from some well-known AI agents:

awk -F'"' '{print $6}' access.log \
  | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|Amazonbot|meta-external[a-z]+|Bytespider|CCBot' \
  | sort | uniq -c | sort -rn

It's a useful first look, but it has real limits: it trusts the user agent, so impostors count as the real thing; it only knows the names you list; and it says nothing about whether the agents succeeded.

What to measure

1. Who's visiting. Requests by agent and by kind: training crawlers, AI search, assistant fetchers and browser agents. The mix matters more than the total. Assistant fetchers are people asking about you right now.
2. Who's real. Verify each agent against its operator's published IP ranges or reverse DNS, and separate verified, spoofed and "can't tell". See how to tell if an AI crawler is real. Without this, scrapers inflate the numbers.
3. What they're reading. The pages agents request most. Are they reaching products, pricing and policies, or stuck on the home page and old URLs?
4. Whether they're blocked. The share of agent requests that get 403, 429 or challenge responses. A rise usually means a bot protection change caught good agents. See bot protection and CAPTCHAs.
5. Errors. 404s for agents often point to old URLs that AI models learned and still request. Redirect them.
6. Trends. Week over week, by agent. New agents appear often, and a sudden drop from one usually means something on your side changed.

Privacy

Logs contain IP addresses, which are personal data for human visitors. You only need the full detail for bots and agents. Hash or drop IP addresses for human traffic, keep agent records separately, and set a retention period that matches your privacy policy.

Measuring traffic tells you who's coming. To find out whether they succeed, test the journeys that matter. See how to make checkout work for AI shopping agents.

What llms.txt is

llms.txt is a proposed standard, first published by Jeremy Howard of Answer.AI in 2024. The idea is simple: put a plain-text Markdown file at /llms.txt on your domain, the same way /robots.txt lives at the root. Where robots.txt tells crawlers what they may visit, llms.txt tells AI models and agents what is worth reading.

It's useful because AI agents have limited attention. A typical page is mostly navigation, scripts, tracking and layout. An agent asked "what's the return policy at this store?" has to dig through all of that. A good llms.txt hands it the answer's location in one request.

The format

The file is ordinary Markdown with a light structure, in this order:

1. An H1 with your name. This is the only required part.
2. A blockquote summary: one or two sentences on what you do and who for.
3. Optional paragraphs with anything an agent should know up front, such as where you ship or what you don't sell.
4. H2 sections of links, each a Markdown list item: - [Page name](URL): what it's for.
5. An "Optional" section for links an agent can skip if it's short on space.

A complete example

Here's an llms.txt for a fictional online store:

# Northwind Coffee
> Northwind Coffee roasts and sells specialty coffee beans, ground coffee and
> brewing equipment online, shipping to the US and Canada.
Orders over $40 ship free in the US. We don't sell gift cards or wholesale online;
wholesale enquiries go through the contact page.
## Shop
- [All coffee](https://northwind.example/collections/coffee): Every bean we sell, with roast level, origin and price
- [Subscriptions](https://northwind.example/subscriptions): Recurring deliveries every 2, 4 or 6 weeks; pause or cancel any time
- [Brewing equipment](https://northwind.example/collections/equipment): Grinders, kettles and pour-over kits
## Help and policies
- [Shipping](https://northwind.example/pages/shipping): Rates, delivery times and countries we ship to
- [Returns](https://northwind.example/pages/returns): 30-day returns on unopened items and equipment
- [FAQ](https://northwind.example/pages/faq): Grind sizes, freshness, subscriptions and account questions
- [Contact](https://northwind.example/pages/contact): Email and live chat hours
## Optional
- [Our story](https://northwind.example/pages/about): Who we are and how we source our beans
- [Brewing guides](https://northwind.example/blog/guides): Step-by-step guides for each brewing method

You can also see our own llms.txt.

What to include

Think about the questions people ask agents about businesses like yours, and link to the page that answers each one.

If you offer agents a direct way in, such as an API or an MCP server, list it with a link to its documentation. AgentScore looks for an MCP server mentioned in llms.txt as one of the signals that a site supports agents directly.

Common mistakes

• Serving HTML. Some site builders return your home page or a 404 page for any unknown URL. Open /llms.txt in a browser and make sure you see plain text, not a web page.
• Writing marketing copy. "The world's most loved coffee experience" tells an agent nothing. Say what you sell, where, and on what terms.
• Linking to everything. It's a guide, not a sitemap. Twenty to forty well-described links beats five hundred bare ones. Your sitemap.xml already lists every page.
• Leaving out descriptions. The text after each link is what lets an agent pick the right page without opening them all.
• Letting it go stale. Dead links and old prices are worse than no file. Review it whenever you change your navigation or policies.
• Linking to pages agents can't read. If a linked page only shows content after JavaScript runs, or sits behind a CAPTCHA, the link won't help. AgentScore checks for both.

How to publish it

1. Write the file in any text editor and save it as llms.txt.
2. Upload it so it's served at the root of your domain: https://yourdomain.com/llms.txt. On most hosts this means putting it in the same folder as robots.txt. On Shopify, WordPress and other platforms, apps and plugins can serve it for you.
3. Check it's served as plain text with a 200 status, and that your bot protection doesn't challenge AI agents that request it.
4. Run AgentScore. The "llms.txt guide for AI" check will pass once the file is live.

What about llms-full.txt?

Some sites, mostly documentation sites, also publish /llms-full.txt: the full text of their key pages in one Markdown file, so an agent can load everything at once. It's a community convention rather than part of the proposal. It's worth it for developer docs; for most stores and marketing sites a good llms.txt is enough.