# Ghost Agent Labs: every guide and blog post > Ghost Agent Labs makes websites work for AI agents. It tests whether AI agents can read, navigate, and complete tasks on a website, and shows how to fix what breaks. It is a Ghost Inspector product. Key facts: - Who it's for: teams whose website drives revenue, especially e-commerce, growth and SEO, and QA and engineering teams. - What it does: shows which AI crawlers, assistants and browser agents visit a site, using server and CDN data rather than a JavaScript tag. It also tests journeys such as checkout and sign-up with real AI agents, and scores agent readiness with prioritized fixes. - What it isn't: a bot blocker. It helps the agents a site wants to succeed, and works alongside existing bot protection. - Price: AgentScore, the agent-readiness test, is free. The score and top issues need no signup; an email address unlocks the full list of fixes. The Ghost Agent Labs platform is in early access with design partners, and self-serve plans are coming with the public launch. - Company: built by the team behind Ghost Inspector (https://ghostinspector.com/), the automated browser testing service. - Our scanner or Ghost Agents visiting your site: email bot@ghostagentlab.com, or see the AgentScore bot and Ghost Agent pages below. - Markdown: every blog post and article is also available as Markdown by adding `index.html.md` to its URL. llms-full.txt has the full text of all of them in one file. This file has the full text of all 42 guides and 20 blog posts, guides first. For a short guide to the site, see https://ghostagentlab.com/llms.txt. --- ## What is agent readiness? A complete guide > How well AI agents can get into your site, understand it, find their way around and complete tasks: what each part means, what good looks like, and where to start. Published October 8, 2026 by Ghost Agent Labs · Start here · https://ghostagentlab.com/articles/agent-readiness/ ### Key takeaways - Agent readiness is whether AI agents can get into your site, understand it, find their way around and complete tasks for customers. - An agent that can't use your site doesn't complain or call support; it moves on and recommends a site it can use. - Start by checking your robots.txt, asking how your bot protection treats AI agents, and fixing the top three issues AgentScore lists. Agent readiness is how well AI agents can get into your website, understand it, find their way around it, and complete the tasks people send them to do. This guide explains each part, what good looks like, and where to start. ### Why it matters now AI assistants increasingly visit websites on a person's behalf: to answer a question, compare products, check a policy, or buy something. Each of those visits stands in for a customer. If the agent can't use your site, it doesn't complain or call support. It moves on to a site it can use, and recommends that one instead. Agent readiness is not the same as SEO, though they overlap. SEO is about being found and ranked. Agent readiness is about what happens next: whether an agent that has found you can actually do something useful with your site. Not every agent wants the same thing. A crawler collecting pages for AI training is very different from an assistant checking your returns policy for a customer, or a browser agent trying to check out. See [the AI agents that visit your website](https://ghostagentlab.com/articles/kinds-of-ai-agents/), and keep the [agent readiness glossary](https://ghostagentlab.com/articles/agent-readiness-glossary/) handy for the terms in this guide. ### The four parts of agent readiness [AgentScore](https://ghostagentlab.com/agentscore/) measures agent readiness out of 100, across four categories. The weights reflect what stops agents most often in practice. | Category | The question | Points | | --- | --- | --- | | Access | Can agents get in? | 25 | | Readability | Can agents understand your pages? | 25 | | Navigability | Can agents find their way around? | 20 | | Task completion | Can an agent actually get the job done? | 30 | #### 1. Access: can agents get in? Before anything else, the agent has to be allowed through the door. Most access failures are accidental: a robots.txt rule copied from a template, or bot protection that challenges every visitor that isn't a person. - **robots.txt** allows AI assistants and AI search agents. You can still block training crawlers separately. See [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). - **Bot protection** doesn't block AI agents that a person would get through, and can tell real agents from impostors. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/) and [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). - **No CAPTCHA or challenge page** on arrival for ordinary pages. - **An llms.txt file** points agents to your most important pages. See [how to write an llms.txt file](https://ghostagentlab.com/articles/llms-txt/). #### 2. Readability: can agents understand your pages? Many agents read the HTML your server sends and never run JavaScript. If your prices, product details or policies only appear after scripts load, those agents see an empty page. - **Key content is in the HTML**, through server-side rendering or static generation, not only added by JavaScript. See [why AI agents can't see JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). - **Structured data** (schema.org) describes products, prices, availability and your business. See [product data AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). - **Clear titles, headings and image descriptions** on every important page. - **A valid sitemap** that lists the pages you want found. #### 3. Navigability: can agents find their way around? Browser agents find their way around a page much as a screen reader does: by the names and roles of buttons, links and form fields. Much of agent readiness here is simply good accessibility. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). - **Every button, link and form field has a name** an agent can read. An icon with no label is invisible. - **Cookie banners and pop-ups** can be dismissed and don't cover the content. - **Menus open on click or focus**, not only on hover. - **Page landmarks** such as header, navigation, main and footer let agents find what they need quickly. #### 4. Task completion: can an agent get the job done? This is the part that matters most, and the only way to measure it is to try. A site can pass every technical check and still lose agents at the size selector or the checkout button. - **Online stores:** an agent can search, find a product, choose options and add it to the cart. See [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/). - **Software companies:** an agent can find pricing and reach the sign-up form. - **Service businesses:** an agent can find prices, availability and a way to book. The free AgentScore scan doesn't test this yet: it marks Task completion “not tested” and works out your score from the other three categories. [Ghost Agents](https://ghostagentlab.com/ghost-agent/) test it today, running the journeys you choose on a schedule, with a step-by-step replay of each run, and alerting you when one breaks. ### How to measure your agent readiness 1. **Run AgentScore.** It's free, takes under a minute, and checks Access, Readability and Navigability. Task completion isn't part of the free scan yet. 2. **Look at your agent traffic.** Your analytics probably can't see agents, because most never run the JavaScript tag. Server or CDN logs can, and Ghost Agent Labs reads them for you. See [how to measure agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). 3. **Test your money journeys.** Checkout, sign-up and booking are the journeys where an agent failure costs you a customer. ### A 30-minute starter checklist > 1. Open `/robots.txt` and make sure it doesn't block AI assistants or AI search agents. > 2. Ask whoever runs your CDN or bot protection how it treats verified AI agents. > 3. View the source of a product or pricing page (not the inspector) and check the price is in the HTML. > 4. Publish an llms.txt file with links to your key pages. > 5. Tab through your header with the keyboard. If you can't reach the menu or the cart, neither can many agents. > 6. Run [AgentScore](https://ghostagentlab.com/agentscore/) and fix the top three items it lists. ### Common questions #### Won't letting agents in mean letting scrapers in? No. Agent readiness is about letting in the agents you want, and the main ones can be verified. Keep blocking unknown and abusive bots; just make sure your rules don't catch AI assistants that are bringing you customers. #### Do I have to let AI companies train on my content? No. Training crawlers and the agents that fetch pages for people use different names, so you can block one and allow the other. Our [robots.txt guide](https://ghostagentlab.com/articles/robots-txt-ai-agents/) shows how. #### Is this just accessibility? There's a big overlap, especially in navigation, and work on one helps the other. But agent readiness also covers access rules, machine-readable data, and whether an agent can finish a real task. ### Sources and further reading - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (IETF) - [Product](https://schema.org/Product) (schema.org) - [Sitemaps XML format](https://www.sitemaps.org/protocol.html) (sitemaps.org) - [The /llms.txt file](https://llmstxt.org/) (llmstxt.org) --- ## The AI agents that visit your website, and what each one wants > Training crawlers, AI search crawlers, assistant fetchers, browser agents and tool-using agents: what each does, what it needs from your site, and how to treat it. Published October 9, 2026 by Ghost Agent Labs · Start here · https://ghostagentlab.com/articles/kinds-of-ai-agents/ “AI bots” isn't one kind of visitor. Some collect pages to train models, some build AI search, some fetch a page because a person just asked about you, and some open your site in a browser and click through it. This guide explains the five kinds, what each one needs from your site, and how to treat each. ### Why the difference matters Each kind of agent does a different job for a different company, and brings you a different kind of value. A training crawler takes your content and may never send anyone back. An assistant fetching your returns policy is answering a customer's question right now. A browser agent on your checkout page is trying to place an order. Treat them all as “bots” and you get one of two bad outcomes: block everything and disappear from AI answers and purchases, or allow everything without knowing what you've agreed to. The rest of this guide gives you the vocabulary to make a separate decision for each. ### The five kinds at a glance | Kind | Why it visits | How it reads your site | What it's worth to you | | --- | --- | --- | --- | | Training crawler | To collect pages for training AI models | Raw HTML, in bulk | Indirect at best | | AI search crawler | To index pages so AI search can cite and link to them | Mostly raw HTML | Visibility in AI answers | | Assistant fetcher | A person asked an assistant about a specific page or question | Usually raw HTML, one page at a time | A customer asking about you right now | | Browser agent | A person asked an assistant to do something, like book or buy | A real browser: renders the page, clicks and types | A task, often a sale, in progress | | Tool-using agent | To call tools your site offers directly, through MCP or an API | Structured answers from your tools, not pages | The fastest, most reliable route to a task | ### 1. Training crawlers Training crawlers collect large numbers of pages to train AI models. Examples include `GPTBot`, `ClaudeBot` and `CCBot`. They don't act for a particular person, and a visit from one doesn't mean anyone is looking for you. - **What they need:** nothing special. They read your HTML like a search crawler. - **How to control them:** robots.txt. They use their own names, so you can block them without blocking AI search or assistants. See [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). - **The decision:** whether your content may be used for training is a business and legal choice, not a readiness one. AgentScore reports blocked training crawlers but doesn't take points off for them. ### 2. AI search crawlers AI search crawlers, such as `OAI-SearchBot`, `Claude-SearchBot` and `PerplexityBot`, build the index that AI search and assistants draw on when they answer questions and cite sources. They work much like Googlebot does for traditional search. - **What they need:** your key content in the HTML your server sends, clear titles and headings, structured data, and a sitemap. Many don't run JavaScript. See [why AI agents can't see JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). - **How to control them:** robots.txt, and your CDN or bot protection, which has to let them through. - **Why allow them:** if they can't read you, AI answers about your category are written from your competitors' pages. ### 3. Assistant fetchers When someone asks ChatGPT, Claude or Perplexity “what's the return window at this store?” or “compare these two products”, the assistant may fetch the pages it needs there and then. Those requests use names like `ChatGPT-User`, `Claude-User` and `Perplexity-User`. - **What they need:** fast, readable pages with the facts in plain text: prices, stock, shipping and returns. See [shipping, returns and FAQ pages AI assistants can quote](https://ghostagentlab.com/articles/policy-pages-ai-assistants/). - **How to control them:** some operators treat these fetches like a person clicking a link and say robots.txt may not apply. Stopping them completely means a rule at your CDN or server. - **Why allow them:** every one of these visits stands in for a customer asking about you. These are usually the visits you want most. ### 4. Browser agents Browser agents go further: they open your site in a real browser and use it the way a person would, to search, choose a size, fill in a form or reach checkout. Several AI assistants now offer an agent mode that does this on a person's behalf, and businesses are starting to use them too. - **What they need:** buttons, links and form fields with clear names, pop-ups they can close, no CAPTCHA in the way, and a checkout that doesn't force an account. They find their way around much as a screen reader does. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/) and [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/). - **How to recognize them:** often you can't from the name alone, because many use an ordinary browser user agent. Some operators now sign their requests with [Web Bot Auth](https://ghostagentlab.com/articles/verify-ai-crawlers/) so sites can tell who they are. - **Why they matter:** they're the agents that complete tasks, so they're where agent readiness turns into revenue, and where a broken button or a bot challenge costs you an order. ### 5. Tool-using agents Instead of reading pages, some agents call tools a site offers them directly: “search products”, “check stock”, “add to cart”. These tools are offered through the Model Context Protocol (MCP), WebMCP in the browser, or agentic commerce protocols for checkout. - **What they need:** a set of well-described, safe tools, starting with read-only ones such as search and product details. - **Why it's worth planning for:** tools are faster and more reliable than clicking through pages, and few sites offer them yet. See [MCP and WebMCP](https://ghostagentlab.com/articles/mcp-webmcp-for-websites/) and [agentic commerce protocols explained](https://ghostagentlab.com/articles/agentic-commerce-protocols/). - **Where to start:** most businesses get more from fixing the first four kinds first. Tools build on readable pages and clean product data, rather than replacing them. ### How to see which ones visit you Your web analytics probably shows almost none of this. Most agents never run the JavaScript tag analytics depends on, and the ones that do are often filtered out as bots. To see them, look at your server or CDN logs, where every request is recorded with its user agent and IP address. Then check that the visitors claiming to be well-known agents really are: names are easy to fake. 1. **Count by kind, not by bot.** Group visits into the five kinds above, so you can see how many customer-driven fetches you get compared with crawls. 2. **Verify the big names.** Check claimed agents against the IP ranges or signatures their operators publish. See [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). 3. **Watch for errors.** A rise in blocked or failed requests from one kind of agent usually means a rule changed. See [finding the errors AI agents hit](https://ghostagentlab.com/articles/agent-errors-in-logs/). Ghost Agent Labs does this for you from your CDN or server logs. See [how to measure agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). ### A sensible starting policy > - **Training crawlers:** your choice. Block them by name in robots.txt if you don't want your content used for training. > - **AI search crawlers and assistant fetchers:** allow them, in robots.txt and at your CDN, and make sure your key content is in the HTML. > - **Browser agents:** allow verified ones through bot protection, especially on product, cart and checkout pages, and test that they can finish your key journeys. > - **Tool-using agents:** plan for them once the basics work, starting with read-only tools. > - **Everyone else:** keep blocking unknown and abusive bots, and anything that claims a trusted name but fails verification. For a fuller decision guide by business type, see [should you block AI crawlers?](https://ghostagentlab.com/blog/block-or-allow-ai-crawlers/) To see how your site treats each kind today, run [AgentScore](https://ghostagentlab.com/agentscore/): its robots.txt check reads your rules as 21 AI agents would. --- ## Agent readiness glossary: the terms you'll hear, in plain English > Plain-English definitions of the terms behind agent readiness, from assistant fetchers and Web Bot Auth to structured data, the accessibility tree, MCP and AgentScore. Published October 9, 2026 by Ghost Agent Labs · Start here · https://ghostagentlab.com/articles/agent-readiness-glossary/ Agent readiness comes with a lot of new words, and some old ones used in new ways. This glossary explains the terms you'll meet in our guides, in AgentScore reports and in conversations with your developers and vendors, in plain English and with what each one means for your business. ### The agents **AI agent.** Software that uses an AI model to do something on the web for a person or a company: read a page, answer a question, compare products or complete a task. In our guides, “agent” covers everything from crawlers to agents that shop. **Training crawler.** A bot that collects pages to train AI models, such as `GPTBot` or `ClaudeBot`. Blocking it doesn't stop AI assistants visiting for a person. **AI search crawler.** A bot that indexes pages so AI search and assistants can cite and link to them, such as `OAI-SearchBot` or `PerplexityBot`. **Assistant fetcher.** An AI assistant fetching a page in real time because a person asked it something, such as `ChatGPT-User` or `Claude-User`. Each visit stands in for a customer. **Browser agent.** An agent that opens your site in a real browser and clicks, types and scrolls like a person, to book, sign up or buy. Also called a computer-use agent or autonomous agent. **User agent.** The name a visitor sends with every request to say what it is, such as a browser version or `GPTBot`. Easy to fake, so it shouldn't be trusted on its own. For how these differ and how to treat each, see [the AI agents that visit your website](https://ghostagentlab.com/articles/kinds-of-ai-agents/). ### Getting in: access **robots.txt.** A file at the root of your site that tells crawlers and agents which pages they may visit. Well-behaved agents follow it; scrapers ignore it. See [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). **Bot protection.** Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors. Set too strictly, they turn away AI assistants as well as scrapers. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/). **CDN.** Content delivery network: the service that sits in front of your site and delivers it quickly around the world. It's often where bot rules live, and where agent traffic is recorded. See [setting up your CDN for AI agents](https://ghostagentlab.com/articles/cdn-settings-ai-agents/). **CAPTCHA or challenge.** A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page. Agents can't pass them, so one on arrival or at checkout ends the visit. **Rate limit.** A cap on how many requests a visitor can make in a given time. Fair limits stop scrapers without turning away agents. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). **Verification.** Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature. See [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). **Impostor (spoofed agent).** A request that uses a trusted agent's name but fails verification. Usually a scraper trying to get past bot protection. **Forward-confirmed reverse DNS.** A verification check that an IP address's hostname belongs to the operator, and that the hostname points back to the same address. **Web Bot Auth.** A new standard in which agents sign their requests cryptographically, so sites can prove who sent them. Our own scanner and Ghost Agents sign theirs. See [why we sign every request](https://ghostagentlab.com/blog/signed-requests/). **llms.txt.** A short Markdown file at `/llms.txt` that tells AI what your business does and links to your key pages. An emerging standard. See [how to write an llms.txt file](https://ghostagentlab.com/articles/llms-txt/). **Guest checkout.** Letting shoppers buy without creating an account. An agent buying for someone can't easily sign up for them. See [guest checkout](https://ghostagentlab.com/articles/guest-checkout-ai-agents/). ### Understanding pages: readability **Raw HTML.** The page exactly as your server sends it, before any JavaScript runs. It's all that many crawlers and assistant fetchers ever see. To look at it, use your browser's “View page source”, not the inspector. **Server-side rendering (SSR) and static generation.** Ways of building pages so the content is already in the raw HTML, rather than added afterwards by JavaScript. See [why AI agents can't see JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). **Structured data (schema.org, JSON-LD).** Machine-readable facts in your page code, such as a product's name, price and stock, written in the shared schema.org vocabulary, usually as JSON-LD. Agents can read it without guessing. See [product data AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). **XML sitemap.** A file listing the pages you want found, so crawlers and agents don't depend on following links. See [XML sitemaps for AI agents](https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/). **Canonical URL.** The one address you name as the main version of a page that can be reached in several ways. It stops agents treating duplicates as different products. See [canonical URLs](https://ghostagentlab.com/articles/canonical-urls-ai-agents/). **Alt text.** A short text description of an image. Agents can't see pictures, so this is how they learn what a product photo shows. See [alt text for AI agents](https://ghostagentlab.com/articles/image-alt-text-ai-agents/). ### Finding the way: navigability **Accessibility tree.** The structured view of a page that screen readers and many browser agents use: its headings, links, buttons and form fields, with their names. If something isn't in it, many agents can't use it. See [what an AI agent sees](https://ghostagentlab.com/blog/what-an-ai-agent-sees/). **Accessible name.** The name a button, link or field has in the accessibility tree, from its text, label or `aria-label`. An icon-only button with no name is invisible to agents. **Landmarks.** Labels for the main regions of a page, such as header, navigation, main content and footer, that let agents skip straight to what they need. **Overlay.** Anything that covers the page, such as a cookie banner, newsletter pop-up or chat window. If an agent can't close it, the journey ends there. See [cookie banners and pop-ups](https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/). **WCAG.** The Web Content Accessibility Guidelines, the standard for making sites usable by people with disabilities. Much of it helps agents too. See [accessibility work is agent readiness work](https://ghostagentlab.com/blog/accessibility-is-agent-readiness/). ### Getting it done: tasks and commerce **Journey.** A task a visitor sets out to complete on your site, such as finding a product and reaching checkout, or booking a call. The journeys that make you money are the ones to test first. **Task completion.** Whether an agent can actually finish a journey. It's the AgentScore category worth the most points, and the only way to measure it is to try. See [how to test your key journeys](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). **MCP (Model Context Protocol).** A standard way to give AI agents direct tools for your site, such as search, product details and cart, instead of making them click through pages. **WebMCP.** A proposal for offering those same tools from inside your web pages, so a browser agent can call them while it's on your site. See [MCP and WebMCP](https://ghostagentlab.com/articles/mcp-webmcp-for-websites/). **Agentic commerce.** Purchases that an AI agent makes for a shopper. Protocols such as the Agentic Commerce Protocol, the Universal Commerce Protocol and AP2 aim to make that safe for buyer, store and payment provider. See [agentic commerce protocols explained](https://ghostagentlab.com/articles/agentic-commerce-protocols/). ### Measuring it **Agent readiness.** How well AI agents can get into your site, understand it, find their way around it and complete tasks on it. See [what is agent readiness?](https://ghostagentlab.com/articles/agent-readiness/) **AgentScore.** Our free score out of 100 for agent readiness, across Access (25 points), Readability (25), Navigability (20) and Task completion (30). The free scan marks Task completion “not tested” and scores the other three. See [how AgentScore works](https://ghostagentlab.com/blog/how-agentscore-works/). **Check.** One thing AgentScore tests, such as “Buttons and links have names”, reported as a pass, warning or fail, with who usually fixes it. **Ghost Agent.** A real AI agent we send through a journey on your site, on a schedule, to see whether agents can complete it. You get a step-by-step replay of each run, and an alert when a journey breaks. See [Ghost Agent](https://ghostagentlab.com/ghost-agent/). **Run and pass rate.** A run is one attempt by one agent at a journey. AI agents don't behave the same way every time, so journeys are tried several times, and the pass rate is the share of runs that succeed. **Flaky.** A journey where some runs pass and some fail. Agents can do it, but not reliably, which usually points to a slow page, a pop-up that appears sometimes, or an unclear control. **Agent traffic.** Visits from AI agents of every kind. Most never run your analytics tag, so it's measured from server or CDN logs. See [how to measure agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). **AI referral.** A person who arrives at your site by clicking a link in an AI assistant's answer. Unlike agent visits, these do show up in analytics. See [tracking visits and sales from AI assistants](https://ghostagentlab.com/articles/ai-referral-traffic/). **Crawls per AI referral.** How many pages an AI company's agents fetched for every visitor its assistant sent you. A low number means you get traffic back for what they read. To see where your own site stands on each of these, run [AgentScore](https://ghostagentlab.com/agentscore/). It's free and takes under a minute. --- ## robots.txt for AI agents: who to allow and who to block > The three kinds of AI agent, the robots.txt names each one uses, a sensible default, and the most-specific-group rule that catches most people out. Published October 8, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/robots-txt-ai-agents/ ### Key takeaways - You can block AI training crawlers and still let AI search and assistants, which answer customers' questions about you, read your site. - A crawler follows only the most specific group that names it, so adding a group for one agent can quietly drop your other rules. - Open your robots.txt today and check it doesn't block AI search or assistant agents, including a leftover rule that blocks everything. Your robots.txt file may be blocking the AI assistants that send you customers, while doing nothing to stop the bots you actually wanted to keep out. This guide explains the three kinds of AI agent, which robots.txt names each one uses, and how to write rules that let the right ones in. ### Three kinds of AI agent "AI bots" isn't one thing. The major AI companies each run several agents with different jobs and different names, and the right policy for each is different. | Kind | What it does | Examples (robots.txt name) | | --- | --- | --- | | Training crawlers | Collect pages to train AI models | `GPTBot`, `ClaudeBot`, `CCBot`, `Bytespider`, `meta-externalagent` | | AI search crawlers | Index pages so AI search can cite and link to them | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Amazonbot` | | Assistant fetchers | Fetch a page in real time because a person asked about it | `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `meta-externalfetcher`, `DuckAssistBot`, `MistralAI-User` | Some names aren't crawlers at all. `Google-Extended` and `Applebot-Extended` are control tokens: Google and Apple crawl with Googlebot and Applebot as usual, and these names let you say whether that content may be used for their AI models. Blocking them doesn't affect normal search. Lists like this change as companies launch new agents, so check each operator's documentation for the current names. ### A sensible default For most businesses, the agents that matter most are AI search and assistant fetchers, because they answer people's questions about you and link back to you. Training is a separate decision about your content, and it's yours to make. This robots.txt allows everything, which is also what you get with no robots.txt at all: ``` User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml ``` If you'd rather not have your content used to train AI models, but still want to appear in AI search and assistant answers, block only the training names: ``` # Allow AI search and assistants; opt out of AI training User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent Disallow: / User-agent: * Disallow: /cart Disallow: /account Allow: / Sitemap: https://www.example.com/sitemap.xml ``` Listing several `User-agent` lines above one set of rules applies those rules to all of them. ### The rule that catches most people out A crawler follows only the most specific group that matches its name, and ignores the rest. If there's a group for `GPTBot`, GPTBot ignores everything under `User-agent: *`. So in the example above, the training crawlers don't inherit the `/cart` and `/account` rules; they don't need them, because they're blocked from everything. But the reverse mistake is common: ``` User-agent: * Disallow: /checkout Disallow: /admin User-agent: OAI-SearchBot Allow: / ``` Someone added that second group to "make sure" OpenAI's search crawler gets in. Instead, they've told it that `/checkout` and `/admin` are open, because it no longer reads the `*` group. If you add a group for a specific agent, copy into it every rule you want it to follow. ### Other common mistakes - **A leftover `Disallow: /`.** Staging sites are often launched with a robots.txt that blocks everything. Check yours today. - **Blocking every AI name from a copied list.** Long "block all AI" lists usually include the assistant fetchers and AI search crawlers too, which removes you from AI answers as well as training. - **Blocking the pages agents need.** Disallowing `/products` or `/pricing` for all bots, or for `*`, to save crawl budget also hides them from agents. - **Relying on robots.txt to block bad bots.** robots.txt is a request, not a lock. Well-behaved crawlers follow it; scrapers ignore it. Abusive traffic has to be stopped at your CDN or firewall. - **Forgetting your CDN.** A firewall rule or bot setting that challenges "AI crawlers" overrides a friendly robots.txt. The agent never gets far enough to read it. ### A note on assistant fetchers When a person asks an assistant to read a specific page, some operators treat that fetch like a person clicking a link rather than a crawl, and say robots.txt rules may not apply to it. If you need to stop those visits completely, that has to happen at your CDN or server, not in robots.txt. For most businesses, though, these are the visits you want most: each one is a customer asking about you. ### How to check your robots.txt 1. Open `https://yourdomain.com/robots.txt` and read it top to bottom. Note every group and which names it applies to. 2. For each AI search crawler and assistant fetcher in the table above, work out which group it follows, remembering the most-specific-group rule, and whether that group blocks your important pages. 3. Run [AgentScore](https://ghostagentlab.com/agentscore/). Its robots.txt check reads your rules as 21 AI agents would and tells you which assistants and AI search agents are blocked from your home page. Blocked training crawlers are reported but don't cost you points, because that's a legitimate choice. robots.txt is only the first door. Next, make sure your bot protection lets the real agents through, and can tell them apart from impostors using their names. See [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). ### Sources and further reading - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (IETF) - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI) - [Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (Anthropic) - [List of Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) (Google) --- ## How to tell if an AI crawler is real > Anyone can claim to be GPTBot or Googlebot. How to verify AI agents with published IP ranges, reverse DNS and signed requests, and what to do when there's no proof. Published October 8, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/verify-ai-crawlers/ ### Key takeaways - Anyone can send a request claiming to be GPTBot or Googlebot, so trusting the name alone lets impostors straight in. - Treat requests you can't verify as unknown rather than fake, because blocking all of them can shut out legitimate agents. - Ask whether your CDN already verifies bots, and make sure the AI assistants and AI search agents you want are allowed. Anyone can send a request that says it's GPTBot or Googlebot. To let real AI agents in while keeping impostors out, you need to check where a request actually came from. There are three ways to do it, and the right one depends on the operator. ### Why user agents can't be trusted Every request names its sender in the `User-Agent` header, and that name is a promise, not a proof. Scrapers routinely borrow the names of well-known crawlers, because many sites let those crawlers through without a challenge. If your bot rules trust the name alone, you're letting in anyone who copies it. If they distrust it, you're blocking real agents along with the fakes. Verification resolves that. It answers one question: did this request really come from the company it names? ### Method 1: published IP ranges Several operators publish the IP addresses their crawlers use, as machine-readable JSON files. If a request claims to be one of their agents and comes from an address in the list, it's real. | Operator | Agents covered | | --- | --- | | Google | Googlebot, special-case crawlers and user-triggered fetchers (separate lists) | | Microsoft | Bingbot | | Apple | Applebot | | OpenAI | GPTBot, OAI-SearchBot and ChatGPT-User (one list each) | | Perplexity | PerplexityBot and Perplexity-User (one list each) | Each operator links its list from its crawler documentation. A few practical points: - **Refresh the lists often.** Ranges change. Fetch them at least daily, and don't treat a request as fake because it's missing from a list that's days old. - **Match the list to the agent.** An OpenAI address in the GPTBot list doesn't verify a request claiming to be ChatGPT-User. - **Handle IPv6.** Several lists include IPv6 ranges. ### Method 2: forward-confirmed reverse DNS Some operators instead promise that their crawlers' IP addresses resolve to a host name on their own domain. The check has two steps, because reverse DNS on its own can be faked by whoever controls the IP address: 1. **Reverse lookup:** look up the host name for the request's IP address, and check it ends in the operator's domain. 2. **Forward lookup:** look up that host name's IP addresses, and check the original IP is among them. 1. A request arrives User agent says Googlebot, from IP address `66.249.66.1` 2. 1. Reverse lookup `66.249.66.1` → `crawl-66-249-66-1.googlebot.com` Ends in googlebot.com 3. 2. Forward lookup `crawl-66-249-66-1.googlebot.com` → `66.249.66.1` Points back to the same address 4. Verified The request is from Google. If either step fails, it isn’t, whatever its user agent says. *Forward-confirmed reverse DNS, using Google’s own example. The second step matters because whoever controls an IP address can make its reverse lookup say anything.* Google's own example, from a terminal: ``` $ host 66.249.66.1 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. $ host crawl-66-249-66-1.googlebot.com crawl-66-249-66-1.googlebot.com has address 66.249.66.1 ``` The name ends in `googlebot.com` and points back to the same address, so the request is from Google. Domains that operators document for this include: | Agent | Host name ends in | | --- | --- | | Googlebot and other Google crawlers | `googlebot.com`, `google.com` or `googleusercontent.com` | | Bingbot | `search.msn.com` | | Applebot | `applebot.apple.com` | | Amazonbot | `crawl.amazonbot.amazon` | DNS lookups are slow compared with a page request, so cache the result for each IP address rather than checking on every hit. ### Method 3: signed requests (Web Bot Auth) The newest method doesn't depend on IP addresses at all. With [Web Bot Auth](https://blog.cloudflare.com/web-bot-auth/), the agent signs each request with a private key and publishes the matching public key on its own domain. Your CDN or server checks the signature, and a valid one proves the sender, whatever network it came from. That makes it the best fit for browser agents running in the cloud, whose addresses change constantly. It's still early, so check whether your CDN supports it. We explain how it works, and why we sign all of our own agents' requests, in [Why we sign every request our agents send](https://ghostagentlab.com/blog/signed-requests/). ### When there's no proof at all Not every operator publishes IP ranges, a reverse DNS domain, or signing keys for every agent. For those, you can't prove a request is real or fake; you can only weigh the evidence, such as how it behaves and how fast it requests pages. Treat "can't tell" as its own answer, not as "fake". Blocking everything you can't verify will block legitimate agents from operators that simply haven't published proof yet. ### Putting it together 1. **Identify:** match the user agent to a known agent and its operator. 2. **Verify:** use the strongest proof that operator offers: a signature, then published IP ranges, then reverse DNS. 3. **Decide:** let verified agents you want through; challenge or block requests that claim a name but fail verification; and handle "can't tell" with ordinary rate limits rather than a hard block. > **Your CDN may already do this.** Many CDNs and bot-management products maintain lists of verified bots. Check that the AI assistants and AI search agents you want are in the allowed categories, and that "AI crawler" blocking rules aren't catching them by name. > > **Ghost Agent Labs does it for you.** It identifies every agent in your server or edge traffic, checks it against published IP ranges and reverse DNS, and flags impostors, so you can see which agents are real before you decide what to allow. Missing or stale data is reported as "can't tell", never as "spoofed". [Start free](https://app.ghostagentlab.com/signup). ### Sources and further reading - [Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) (Google) - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI) - [About Applebot](https://support.apple.com/en-us/119829) (Apple) - [HTTP Message Signatures for automated traffic Architecture (Web Bot Auth draft)](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/) (IETF) --- ## Bot protection and CAPTCHAs: stop blocking the agents you want > Why bot protection catches AI assistants, how to tell if it's happening, and how to tune your CDN, WAF and CAPTCHAs so good agents get through and bad bots don't. Published October 8, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/bot-protection-ai-agents/ ### Key takeaways - Bot protection often stops the AI assistants answering customers' questions about your products, because those agents look like bots for legitimate reasons. - Never let traffic through on its name alone, and check that any one-click block AI setting doesn't also block assistants and AI search. - Keep product, pricing and policy pages open, and save strict challenges and CAPTCHAs for login, sign-up and checkout. Bot protection exists to keep out scrapers, credential stuffers and fraud. But the same rules often stop the AI assistants trying to answer a customer's question about your products. This guide shows how to tell whether that's happening, and how to tune your defenses so the agents you want get through and the bots you don't still get stopped. ### Why good agents get caught Bot protection, whether it's a CDN feature, a web application firewall (WAF) or a dedicated product, looks for signs that a visitor isn't a person. AI agents show many of those signs, for entirely legitimate reasons: - **They say they're bots.** Honest agents announce themselves in their user agent, and a rule that blocks "bots" or "AI crawlers" by name catches them first. - **They don't run JavaScript.** Many challenges work by running a script in the visitor's browser. Assistant fetchers and AI crawlers usually read only the HTML, so they never pass. - **They come from data centers.** Agents run in the cloud, and cloud IP addresses score as higher risk than home broadband. - **Browser agents look automated.** Even agents that drive a real browser move and type differently from people, and fingerprinting tools notice. - **One-click "block AI" settings.** Several providers offer a single switch to block AI bots. Depending on how it's set up, it can block assistants and AI search along with training crawlers. ### Signs it's happening to you - AI assistants say they "can't access" your site, or describe it from out-of-date information. - Your CDN or firewall logs show 403 (forbidden), 429 (too many requests) or challenge responses for agents like ChatGPT-User, Claude-User or Perplexity-User. - You appear in AI search results much less often than competitors with weaker SEO. The quickest test is [AgentScore](https://ghostagentlab.com/agentscore/). It requests your home page as a normal browser and as five AI agents, including ChatGPT-User, Claude-User, Perplexity-User and OAI-SearchBot, and compares what comes back. It fails the check if an agent is blocked or challenged while the browser gets through, and warns if an agent gets less than half the content the browser does. Separate checks look for CAPTCHAs on your home page, and test whether agents can open your product, pricing and cart pages. ### How to fix it #### 1. Allow verified agents, not names The safe way to let agents in is by verified identity. Most bot management products keep a list of verified bots, checked against the operator's published IP ranges, reverse DNS or signatures, and let you allow categories of them. Allow the categories for AI assistants and AI search, and keep blocking requests that only claim those names. Our guide to [telling real AI crawlers from fakes](https://ghostagentlab.com/articles/verify-ai-crawlers/) explains how verification works. > **Never allow by user agent alone.** A rule like "if the user agent contains GPTBot, skip all checks" is the first thing scrapers exploit. #### 2. Separate training from assistants If you've turned on a "block AI bots" setting, check exactly which agents it covers. If your goal is to opt out of AI training, block only training crawlers, and leave assistant fetchers and AI search crawlers allowed. See [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/) for which names are which. #### 3. Protect actions, not pages Abuse mostly targets actions: logging in, creating accounts, applying discount codes, checking gift card balances, submitting payment. It rarely needs protecting against on pages that only display information. Concentrate strict rules and challenges on: - Login, sign-up and password reset forms - Checkout submission and payment - Gift card, coupon and stock-check endpoints that attackers hammer - Search and APIs, with rate limits rather than outright blocks Keep product, category, pricing, policy and help pages as open as you safely can. Those are the pages agents need to answer questions about you. #### 4. Use rate limits instead of blocks A real assistant fetcher makes a handful of requests because a person asked about you. A scraper makes thousands. A sensible rate limit per IP address or verified agent stops the scraper without punishing the assistant, where a blanket block stops both. #### 5. Replace puzzle CAPTCHAs on key pages AI agents can't solve CAPTCHAs, and aren't meant to. If you need a challenge, use invisible or risk-based challenges that only escalate when traffic looks abusive, and keep them off the pages agents need to read. Never put a CAPTCHA in front of ordinary content on arrival. #### 6. Don't serve agents a different page Some setups don't block agents outright, but quietly serve them a stripped-down page, a placeholder, or an "enable JavaScript" message. To the agent this is as bad as a block, and it's harder to notice. Agents should get the same content a browser does. ### A quick checklist > 1. Run [AgentScore](https://ghostagentlab.com/agentscore/) and look at the bot protection, CAPTCHA and key pages checks. > 2. In your CDN or bot management settings, find the verified bots or AI categories, and confirm AI assistants and AI search are allowed. > 3. Check that any "block AI" setting covers only the agents you mean to block. > 4. Search your firewall rules for user agent matches and make sure none allow access by name alone. > 5. Move strict challenges to login, sign-up and checkout submission, and use rate limits elsewhere. > 6. Re-run AgentScore to confirm the fix. To keep an eye on this over time, watch how often agents are blocked in your traffic. See [how to measure AI agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). ### Sources and further reading - [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) (IETF) - [RFC 6585: Additional HTTP Status Codes](https://www.rfc-editor.org/rfc/rfc6585.html) (IETF) --- ## How to write an llms.txt file > What llms.txt is, what to put in it, a complete example for an online store, and the mistakes that make it useless to AI agents. Published October 8, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/llms-txt/ ### Key takeaways - An llms.txt file is a short plain-text guide that tells AI agents what your business does and where your key pages are. - It's a proposal that not every AI company reads, so treat it as one part of agent readiness, not a replacement for clear pages. - List twenty to forty well-described links to the pages customers ask about, publish it at your domain root, and keep it current. An llms.txt file is a short Markdown guide to your website, written for AI. It tells an agent what your business does and where the pages that matter are, so it doesn't have to work it out from menus, pop-ups and marketing copy. It takes about half an hour to write, and AgentScore checks for it. ### What llms.txt is [llms.txt](https://llmstxt.org/) is a proposed standard, first published by Jeremy Howard of Answer.AI in 2024. The idea is simple: put a plain-text Markdown file at `/llms.txt` on your domain, the same way `/robots.txt` lives at the root. Where robots.txt tells crawlers what they may visit, llms.txt tells AI models and agents what is worth reading. It's useful because AI agents have limited attention. A typical page is mostly navigation, scripts, tracking and layout. An agent asked "what's the return policy at this store?" has to dig through all of that. A good llms.txt hands it the answer's location in one request. > **An honest caveat:** llms.txt is a proposal, not something every AI company has committed to read. Some coding assistants and documentation tools use it today, and it's cheap to add, but treat it as one part of agent readiness alongside clear pages, structured data and a sitemap, not a replacement for them. ### The format The file is ordinary Markdown with a light structure, in this order: 1. **An H1 with your name.** This is the only required part. 2. **A blockquote summary**: one or two sentences on what you do and who for. 3. **Optional paragraphs** with anything an agent should know up front, such as where you ship or what you don't sell. 4. **H2 sections of links**, each a Markdown list item: `- [Page name](URL): what it's for`. 5. **An "Optional" section** for links an agent can skip if it's short on space. ### A complete example Here's an llms.txt for a fictional online store: ``` # Northwind Coffee > Northwind Coffee roasts and sells specialty coffee beans, ground coffee and > brewing equipment online, shipping to the US and Canada. Orders over $40 ship free in the US. We don't sell gift cards or wholesale online; wholesale enquiries go through the contact page. ## Shop - [All coffee](https://northwind.example/collections/coffee): Every bean we sell, with roast level, origin and price - [Subscriptions](https://northwind.example/subscriptions): Recurring deliveries every 2, 4 or 6 weeks; pause or cancel any time - [Brewing equipment](https://northwind.example/collections/equipment): Grinders, kettles and pour-over kits ## Help and policies - [Shipping](https://northwind.example/pages/shipping): Rates, delivery times and countries we ship to - [Returns](https://northwind.example/pages/returns): 30-day returns on unopened items and equipment - [FAQ](https://northwind.example/pages/faq): Grind sizes, freshness, subscriptions and account questions - [Contact](https://northwind.example/pages/contact): Email and live chat hours ## Optional - [Our story](https://northwind.example/pages/about): Who we are and how we source our beans - [Brewing guides](https://northwind.example/blog/guides): Step-by-step guides for each brewing method ``` You can also see [our own llms.txt](https://ghostagentlab.com/llms.txt). ### What to include Think about the questions people ask agents about businesses like yours, and link to the page that answers each one. | Type of site | Link to | | --- | --- | | Online store | Main categories, bestsellers, shipping, returns, sizing, FAQ, contact | | Software company | Product overview, pricing, sign-up, docs, API reference, security, status page | | Local or service business | Services and prices, booking, locations and hours, service area, contact | | Publisher | Sections, subscription options, licensing and permissions, editorial policy | If you offer agents a direct way in, such as an API or an MCP server, list it with a link to its documentation. AgentScore looks for an MCP server mentioned in llms.txt as one of the signals that a site supports agents directly. ### Common mistakes - **Serving HTML.** Some site builders return your home page or a 404 page for any unknown URL. Open `/llms.txt` in a browser and make sure you see plain text, not a web page. - **Writing marketing copy.** "The world's most loved coffee experience" tells an agent nothing. Say what you sell, where, and on what terms. - **Linking to everything.** It's a guide, not a sitemap. Twenty to forty well-described links beats five hundred bare ones. Your sitemap.xml already lists every page. - **Leaving out descriptions.** The text after each link is what lets an agent pick the right page without opening them all. - **Letting it go stale.** Dead links and old prices are worse than no file. Review it whenever you change your navigation or policies. - **Linking to pages agents can't read.** If a linked page only shows content after JavaScript runs, or sits behind a CAPTCHA, the link won't help. AgentScore checks for both. ### How to publish it 1. Write the file in any text editor and save it as `llms.txt`. 2. Upload it so it's served at the root of your domain: `https://yourdomain.com/llms.txt`. On most hosts this means putting it in the same folder as robots.txt. On Shopify, WordPress and other platforms, apps and plugins can serve it for you. 3. Check it's served as plain text with a 200 status, and that your bot protection doesn't challenge AI agents that request it. 4. Run [AgentScore](https://ghostagentlab.com/agentscore/). The "llms.txt guide for AI" check will pass once the file is live. ### What about llms-full.txt? Some sites, mostly documentation sites, also publish `/llms-full.txt`: the full text of their key pages in one Markdown file, so an agent can load everything at once. It's a community convention rather than part of the proposal. It's worth it for developer docs; for most stores and marketing sites a good llms.txt is enough. ### Sources and further reading - [The /llms.txt file](https://llmstxt.org/) (llmstxt.org) --- ## MCP and WebMCP: giving AI agents a direct way into your site > What MCP servers and WebMCP tools are, what they mean for your business, how AgentScore detects them, and how to start safely. Published October 9, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/mcp-webmcp-for-websites/ ### Key takeaways - MCP and WebMCP let AI agents call named actions, such as searching products, instead of clicking through your pages. - Prices, stock and policies your tools return must match your website exactly, or agents will quote figures your checkout won't honor. - Ask your commerce platform about MCP support first, and start with read-only tools for search, product details, pricing and policies. Most AI agents use your website the way a person does: they read pages, click links and fill in forms. That works, but it's slow and it breaks easily. MCP and WebMCP give agents a second way in, a set of named actions such as "search products" or "check order status" that they can call directly. This guide explains both, what they mean for your business, and how to start small. ### Two ways an agent can use your site Today, an AI agent asked to find a waterproof jacket in a medium on your store has to work your pages. It loads the home page, finds the search box, reads the results, opens a product, works out the size picker, and reads the price. Every step depends on your page layout. A redesigned menu or a new pop-up can stop it. The alternative is to tell the agent what it can do and let it ask directly. Instead of driving the search box, it calls a `search_products` action with "waterproof jacket, medium" and gets back a clean list of products, prices and stock. That's the idea behind both MCP and WebMCP. - **Faster.** One request instead of a dozen page loads and clicks. - **More reliable.** Actions don't change when you redesign the page. - **More accurate.** You decide exactly what data comes back, so agents quote the right price and stock. It doesn't replace a usable website. Many agents will keep using your pages for a long time, and the people they hand back to always will. Think of MCP and WebMCP as an extra, faster lane for the agents that support them. ### What MCP is The [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) is an open protocol for connecting AI applications to outside tools and data. Anthropic introduced it in late 2024, and it has since been adopted by many AI assistants, developer tools and software companies. An MCP server is a small service that describes a set of **tools** (actions an agent can call, each with a name, a plain-language description and the inputs it expects) and can also offer **resources** (data it can read, such as a catalog or a policy). An AI application connects to the server, reads the list, and calls the tools it needs to complete a person's request. For a website, a remote MCP server runs alongside your site, usually on your own domain, and talks to the same systems your site does. Typical tools for an online store: | Tool | What it does | Risk | | --- | --- | --- | | `search_products` | Finds products by keyword, category, size or price | Read-only | | `get_product` | Returns price, variants, stock and delivery estimate for one product | Read-only | | `get_policy` | Returns shipping, returns or warranty terms | Read-only | | `get_order_status` | Looks up an order for a signed-in customer | Needs sign-in | | `add_to_cart` | Builds a cart and returns a link for the person to check out | Changes state | Software companies might offer tools to look up plans and pricing or search documentation. Service businesses might offer tools to check availability and request a booking. Commerce platforms are starting to offer MCP servers for the stores they host, so check with yours before building one. If you have a developer team and an existing API, a read-only server is often a small project, because it wraps calls you already make. ### What WebMCP is WebMCP is an early proposal for bringing the same idea into the web page itself. Instead of running a separate server, your page registers tools with the browser through a new JavaScript API, `navigator.modelContext`. An agent working in that browser can then call the page's tools directly instead of clicking through it. It's being developed in the open at the W3C's Web Machine Learning Community Group, with engineers from Google and Microsoft among those working on it. Community group work is incubation, not a finished standard, and browsers don't ship WebMCP by default yet. Expect the details to change. Two features make it interesting for websites: - **It reuses your page code.** The tool can call the same functions your search box or add-to-cart button already calls. - **It works with the person's session.** Because it runs in the page, it can use the cart and sign-in the person already has, without separate API credentials. A tool registered in a page looks roughly like this, guarded so it does nothing in browsers without the API: ``` if ("modelContext" in navigator) { navigator.modelContext.registerTool({ name: "search_products", description: "Search Northwind Outdoor products by keyword, size and maximum price.", inputSchema: { type: "object", properties: { query: { type: "string" }, size: { type: "string" }, maxPrice: { type: "number" } }, required: ["query"] }, async execute({ query, size, maxPrice }) { const results = await searchCatalog({ query, size, maxPrice }); // your existing search return { content: [{ type: "text", text: JSON.stringify(results) }] }; } }); } ``` The proposal also describes a declarative form: adding a `toolname` attribute (with a description) to an existing `
`, so a search or contact form becomes a tool without new JavaScript. Attribute names may change as the proposal develops. ### MCP or WebMCP? | | MCP server | WebMCP | | --- | --- | --- | | Where it runs | A service on your domain | Inside your web pages | | Who can use it | AI applications that connect to MCP servers | Agents working in a browser that supports it | | Maturity | Published specification, widely used | Early proposal, not shipped by default | | Sign-in | Needs its own authorization for personal data | Uses the person's existing session | | Best first step | Read-only catalog, pricing and policy tools | Mark up your search form, then experiment | For most businesses, an MCP server is the practical choice today, and WebMCP is worth a small experiment so you're ready if browsers adopt it. ### What AgentScore looks for The [AgentScore](https://ghostagentlab.com/agentscore/) check "Agents can use your site through MCP or WebMCP" is part of the Access category. It passes if it finds any of these: - **An MCP server description** at `/.well-known/mcp.json` or `/.well-known/mcp`. There's no settled standard for advertising an MCP server yet, and proposals for a well-known server description are still being discussed, so we look in the most likely places. - **A link in your home page**: a `` or `` whose `rel` or `type` mentions MCP. - **An MCP server listed in your llms.txt**, with its URL. See [how to write an llms.txt file](https://ghostagentlab.com/articles/llms-txt/). - **WebMCP tools registered in your home page.** Because browsers don't ship WebMCP yet, the scanner provides a stand-in `navigator.modelContext` before your scripts run, so pages that check for the API register their tools as they would in a supporting browser. We record the tool names. - **Forms marked up as WebMCP tools** with a `toolname` attribute. If none is found, the check is a warning rather than a failure. Agents can still use your pages; they just have to do it the slow way. It carries less weight than the checks that decide whether agents can get in at all, such as bot protection and robots.txt. > **Advertising a server isn't the same as it working.** AgentScore confirms you've told agents where to find your tools. It doesn't call them. Test your tools with an MCP client before you publish the link, and keep them working as your catalog changes. ### How to start, safely 1. **Ask your platform first.** If your commerce platform or CMS offers an MCP server or WebMCP support, turning it on is far cheaper than building one. 2. **Start read-only.** Search, product details, pricing, stock and policies. These answer most agent questions and can't change anything. 3. **Write tool descriptions for a reader who has never seen your site.** The description is how an agent decides which tool to call. "Search products by keyword, size and maximum price; returns name, price, stock and URL" beats "Search". 4. **Return the same facts your pages show.** Prices, stock and policies must match the website exactly, or agents will quote figures your checkout won't honor. 5. **Keep people in charge of money.** For anything that changes state, such as building a cart, return a link the person opens to review and pay. Leave fully agent-led checkout to the [agentic commerce protocols](https://ghostagentlab.com/articles/agentic-commerce-protocols/) designed for it. 6. **Protect it like an API.** Rate limits, logging and authorization for anything personal. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). 7. **Advertise it.** List the server in llms.txt with a link to its documentation, and add a well-known description or a `` in your page head. ### What it means for the business An MCP server or WebMCP tools won't bring traffic on their own. They make each agent visit more likely to end well: the right product found, the right price quoted, the cart built. Because support among AI applications is still growing, the best order of work is the usual one. Make sure agents can get in and read your pages first, then add a direct lane for the agents that can use it. Not sure where you stand on the basics? Start with [what agent readiness means](https://ghostagentlab.com/articles/agent-readiness/), then run AgentScore to see every Access check, including this one. ### Sources and further reading - [What is the Model Context Protocol (MCP)?](https://modelcontextprotocol.io/) (Model Context Protocol) - [WebMCP (Draft Community Group Report)](https://webmachinelearning.github.io/webmcp/) (W3C Web Machine Learning Community Group) --- ## Agentic commerce protocols explained > What ACP, UCP and AP2 are for, what they mean for your store, what to ask your platform, and what AgentScore can and can't detect. Published October 9, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/agentic-commerce-protocols/ ### Key takeaways - Agentic commerce protocols let an AI assistant buy from your store for a person, without anyone filling in your checkout pages. - Assistants will quote whatever price, stock and delivery date your data returns, so mistakes reach many customers at once. - Fix the website journey and product data first, then ask your platform and payment provider which protocols they support. Agentic commerce protocols let an AI assistant buy from your store on a person's behalf without filling in your checkout pages. The person says "yes, buy it" in the assistant, and the order arrives in your systems like any other. Several protocols have been announced in a short time, by different companies, and the names are easy to mix up. This guide explains what each one is for, what it means for your store, and what to ask your platform. ### Why protocols, when agents can use checkout pages? A browser agent can shop through your website, and for now that's how most agents buy. We cover that journey in [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/). But checkout pages were built for people. An agent filling them in has to guess at fields, cope with pop-ups and CAPTCHAs, and usually hand back to the person to pay. A protocol replaces the page-filling with a structured conversation between the assistant and your store: - **The assistant asks** for a product, variant, quantity and delivery address. - **Your store answers** with the real total, shipping options, tax and any errors, in data rather than on a page. - **The person approves** the purchase inside the assistant. - **Payment is passed** in a form that proves the person authorized it, without the assistant seeing raw card numbers. - **Your store creates the order** and stays responsible for fulfillment, returns and customer service. ### The protocols you'll hear about This is a young and fast-moving area. The descriptions below are deliberately brief, and the protocols' own documentation is the place for detail. Programs, eligibility and regions change often. | Protocol | Who's behind it | What it covers | | --- | --- | --- | | Agentic Commerce Protocol (ACP) | OpenAI and Stripe | Checkout between an AI assistant and a merchant. It's the protocol behind buying inside ChatGPT, and it has been published as an open specification. | | Universal Commerce Protocol (UCP) | Google, with industry partners | A broader standard for agents to discover what a store supports and complete checkout. A store publishes a profile at `/.well-known/ucp`. | | Agent Payments Protocol (AP2) | Google, with payments industry partners | The payment step specifically: a way to prove that a person authorized an agent to make a purchase, designed to work with different payment methods and alongside other protocols. | Card networks and payment providers have also announced their own programs for payments made by agents. Most of these sit underneath the protocols above, at the payment layer, rather than competing with them. > **Checkout and payment are different jobs.** ACP and UCP are about the shopping conversation and creating the order. AP2 is about trusting the payment. A store may end up supporting more than one, often without knowing, because the platform and payment provider handle it. ### What it means for your store - **Assistants become a sales channel.** Supporting a protocol can let a person buy from you without leaving the assistant they asked. That puts you in front of shoppers at the moment they decide. - **You stay the merchant.** In the protocols announced so far, the store keeps the order, the customer relationship and responsibility for fulfillment. Check the terms of any program you join. - **Your data has to be right.** An assistant will quote the price, stock and delivery date your feed or API returns. Mistakes become disappointed customers at scale. See [product data AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). - **Fraud and risk rules need a second look.** Orders arrive without the usual browser signals. Ask your payment provider how it scores them. - **Returns and support still come to you.** Make sure your team can see which orders came through an assistant. ### How stores get support Very few stores will build a protocol integration themselves. Support usually arrives in one of three ways: 1. **Through your commerce platform.** Hosted platforms are adding protocol support for the stores they run, sometimes as a setting you switch on. 2. **Through your payment provider.** Some providers handle the agent-facing checkout and payment for you. 3. **Through an AI platform's merchant program.** Some assistants ask merchants to apply, and review product data and policies before listing them. If you run a custom stack, the specifications are public, but budget for real engineering: order creation, tax, shipping, inventory and payment all have to behave exactly as they do on your site. ### Questions to ask your platform and payment provider - Which agentic commerce protocols do you support today, and which are planned? - Is it on by default, or do we need to enable it or apply? - Which product data does it use, and how do we keep it in sync with the website? - How are agent orders marked in our order system and analytics? - How are fraud screening, chargebacks and refunds handled for agent orders? - Can we limit it to certain products, countries or order values while we learn? ### What AgentScore checks For stores, [AgentScore](https://ghostagentlab.com/agentscore/) includes the Access check "Agents can check out through an agentic commerce protocol". It's skipped for sites that aren't stores. It can only see what a store publishes openly. Today that means a Universal Commerce Protocol profile: the scanner requests `/.well-known/ucp`, and if it gets back a JSON document, the check passes. The Agentic Commerce Protocol is different. Integrations are set up privately between each store and each AI platform, so there's nothing on your site for a scanner to find. If you support ACP but not UCP, this check will still show a warning. That's a limit of what can be seen from outside, not a problem with your store. The advice in the report names both protocols. A warning here costs little. The check has a small weight in the Access category, because most agents still shop through the website, and that's where the checks that decide whether agents can get in and get through checkout matter more. Those include bot protection, CAPTCHAs at the cart and checkout, and [guest checkout](https://ghostagentlab.com/articles/guest-checkout-ai-agents/). ### Where to start 1. **Fix the website journey first.** Make sure agents can get in, read your product pages and reach checkout. Every protocol relies on the same product data and policies. 2. **Get your product data in order.** Accurate prices, variants, stock and delivery times in structured data and in any product feeds you publish. 3. **Ask the questions above** of your platform and payment provider, and turn on what's available. 4. **Watch how agents arrive.** Track agent visits and the orders that come from assistants, so you can tell whether a protocol is earning its keep. See [how to measure AI agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/) and [tracking visits and sales from AI assistants](https://ghostagentlab.com/articles/ai-referral-traffic/). Agentic commerce protocols will keep changing for a while. The stores best placed to benefit are the ones whose websites, data and policies already work for agents, because that's what every protocol is built on. ### Sources and further reading - [Agentic Commerce Protocol](https://www.agenticcommerce.dev/) (Agentic Commerce Protocol) - [Universal Commerce Protocol specification overview](https://ucp.dev/specification/overview/) (Universal Commerce Protocol) - [AP2: Agent Payments Protocol documentation](https://ap2-protocol.org/) (Agent Payments Protocol) --- ## Guest checkout: why AI shopping agents need it > Why forced accounts stop AI agents at the last step, how AgentScore checks for guest checkout, and how to offer it without losing accounts. Published October 9, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/guest-checkout-ai-agents/ ### Key takeaways - AI agents can't create accounts for shoppers and usually can't sign in as them, so forced accounts stop orders at the last step. - A cart page that says shoppers must sign in can undo a working guest checkout and lose a sale the agent nearly finished. - Check whether your platform's checkout setting requires accounts, since making them optional may be the quickest win on your report. An AI agent can find your product, choose the size and fill the cart, then stop dead at a page that says "Sign in to continue". It can't create an account for the person it's shopping for, and it usually can't sign in as them either. Guest checkout removes that dead end. This guide explains why it matters, how AgentScore checks for it, and how to offer it without giving up on customer accounts. ### Why accounts stop agents A forced account is a known source of abandoned checkouts with people. With AI agents it's usually worse, for practical reasons: - **Agents can't sign up on someone's behalf.** Creating an account means choosing a password, agreeing to terms and often confirming an email address. Well-behaved agents won't invent a password or accept terms for a person, and they can't open the person's inbox to click a confirmation link. - **Signing in is a handoff.** Browser agents generally stop at sign-in and pass control back to the person. Some people will finish; many won't, especially if they asked the assistant precisely so they wouldn't have to. - **Sign-in pages attract bot defenses.** Login forms are a favorite target for credential-stuffing attacks, so they often carry the strictest CAPTCHAs and bot rules. An agent sent there meets the hardest wall on your site. - **The failure happens at the last step.** The agent has already done the expensive work: searching, comparing, choosing. Losing the order at sign-in wastes all of it, and the assistant may recommend a store where it could finish instead. Guest checkout fixes all four. The agent, or the person it hands back to, enters an email address and a delivery address and carries on to payment. ### What good guest checkout looks like - **Guest is a clear choice, in words.** A button or link that says "Check out as guest" or "Continue as guest", not an unlabelled icon or a link hidden below a sign-in form. - **Guest is the default path**, or at least equal to sign-in. Many stores go straight to the contact and delivery form and offer sign-in as a link at the top. - **It asks only for what the order needs:** email, name, delivery address, phone if your carrier requires it. Every field is labelled and uses the right `autocomplete` value. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). - **No password field** on the guest path. If you want to offer an account, offer it after the order. - **No CAPTCHA on every checkout.** Use risk-based checks that challenge only suspicious orders. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/). - **The cart doesn't say otherwise.** A cart page that reads "Sign in to check out" undoes a perfectly good guest checkout behind it. ### Keeping the benefits of accounts Accounts matter to most stores: repeat purchases, order history, loyalty and marketing consent. Guest checkout doesn't mean giving those up. It means moving the ask to a moment when it doesn't cost you the sale. | Instead of | Try | | --- | --- | | Requiring an account before checkout | Offering "Save your details for next time" on the order confirmation page | | Asking for a password at checkout | Emailing a link to set up an account from the order | | Hiding order tracking behind sign-in | Order tracking by order number and email address | | Requiring sign-in for loyalty points | Matching guest orders to existing accounts by email, then awarding points | | An account wall for members-only prices | Showing the public price to guests and the member price once signed in | Some stores genuinely need accounts: trade-only wholesale, age-restricted products, prescription items, or subscription services where the account is the product. If that's you, say so clearly on product pages and in your [llms.txt](https://ghostagentlab.com/articles/llms-txt/), so agents can tell the person up front rather than discovering it at checkout. ### How AgentScore checks for guest checkout The Access check "Shoppers can check out without an account" applies to stores. The scanner is read-only: it loads your cart and checkout pages, but never submits a form or adds anything to a cart. It looks at what a shopper would see: - **Pass** if the cart or checkout offers a guest option in words, such as "Check out as guest", "Continue as guest" or "No account needed", or if the checkout asks for contact and delivery details with no password field. - **Fail** if the checkout redirects to a sign-in page, asks for a password with no guest option, or the cart says shoppers must sign in to check out. - **Not tested** if the checkout only opens with items in the cart. Because the scanner never adds items, many checkouts send it back to an empty cart. In that case the check is skipped and doesn't count against your score, rather than guessing. It carries a moderate weight in the Access category, alongside the related check that AI agents aren't blocked or shown a CAPTCHA at the cart and checkout. If your checkout couldn't be tested, a Ghost Agent can walk the full journey with items in the cart. See [how to test your key journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). ### How to check your own store in five minutes 1. Open your store in a private browsing window, so you're signed out. 2. Add any product to the cart and open the cart page. Read it as an agent would: does any text say you must sign in? 3. Press the checkout button. Note where you land: a sign-in page, a choice between sign-in and guest, or straight into a contact and delivery form. 4. If there's a guest option, make sure it's a real button or link with words on it, and that it works with the keyboard alone. 5. Go through to the payment step without creating an account. Note every field, pop-up and challenge on the way. > **Platform settings often decide this.** Most commerce platforms have a single setting for whether accounts are required, optional or disabled at checkout. If yours is set to required, changing it may be the quickest win on your whole AgentScore report. Check with your platform admin before redesigning anything. ### Guest checkout and agentic commerce Assistants that buy through an [agentic commerce protocol](https://ghostagentlab.com/articles/agentic-commerce-protocols/) don't use your checkout pages, so they bypass this problem. But most agents still shop through the website, and the people they hand back to always do. Guest checkout is the simplest change that makes both work, and it helps human shoppers in a hurry just as much. For the rest of the journey, from search to the payment handoff, see [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/). Then run [AgentScore](https://ghostagentlab.com/agentscore/) to see where your store stands. ### Sources and further reading - [HTML Standard: Form control infrastructure (autofill)](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html) (WHATWG) --- ## Rate limits for AI agents: fair limits that don't turn customers away > How to set rate limits that stop scrapers but not AI assistants: 429 with Retry-After, per-agent limits for verified agents, and what to watch. Published October 9, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/rate-limits-ai-agents/ ### Key takeaways - Badly set rate limits can turn away AI assistants looking up your products, partly because many agents share the same cloud addresses. - Answering with a block or CAPTCHA makes many agents give up, while a too many requests response with a wait time asks them to retry. - Give verified AI assistants generous limits, and put your strictest limits on logins, coupons and search rather than product pages. Rate limits protect your site from scrapers and floods of automated traffic. Set badly, they also turn away the AI assistants that are looking up your products for a customer. This guide explains how to set limits that stop abuse without blocking the agents you want, and how to say "slow down" in a way agents understand. ### Why limits catch good agents Most rate limits count requests per IP address. That made sense when one address meant one visitor. It works less well for AI agents: - **Agents share addresses.** Assistant fetchers and browser agents run in cloud data centers, so many different people's requests can arrive from a small pool of addresses. A per-IP limit treats them as one heavy user. - **Agent visits come in bursts.** A person asks an assistant to compare three jackets, and it opens a category page, three product pages and a returns policy within seconds. That's a normal visit, but it can look like a spike. - **Crawlers and assistants get lumped together.** A training crawler reading your whole catalog and an assistant answering one shopper's question have very different value to you, but a single "bots" limit treats them the same. - **Blocks look like limits.** Many setups answer a rate-limited request with a 403 Forbidden or a challenge page. The agent can't tell it should wait, so it reports your site as unavailable. ### Say "slow down" properly: 429 and Retry-After HTTP has a status code made for this. **429 Too Many Requests**, defined in [RFC 6585](https://www.rfc-editor.org/rfc/rfc6585), tells the client it has sent too many requests. Pair it with a **Retry-After** header, defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110), saying how many seconds to wait: ``` HTTP/1.1 429 Too Many Requests Retry-After: 30 Content-Type: text/plain Too many requests from this client. Please wait 30 seconds and try again. ``` Well-behaved crawlers and agents treat this as an instruction, not a door slammed shut. Google's crawler documentation, for example, says Googlebot slows down when it gets 429, 500 or 503 responses. Compare the alternatives: | Response | What the agent concludes | | --- | --- | | 429 with Retry-After | Wait, then try again. The site is fine. | | 429 without Retry-After | Back off, but guess for how long. | | 503 Service Unavailable | The site is having problems. Fine for real outages, misleading for rate limits. | | 403 Forbidden | I'm not allowed here. Many agents give up and say so. | | 200 with a challenge or "access denied" page | Confusing. Some agents read the challenge text as your content. | Keep the 429 lightweight: a short plain-text message, no heavy page, and never a CAPTCHA, which agents can't solve. There's also an IETF draft for `RateLimit` headers that tell clients their remaining allowance before they hit the limit. It's still a draft, so treat it as a bonus for API clients, not a replacement for Retry-After. ### Limit by who, not just by address The fairest limits are set per agent, based on verified identity. Most CDNs and bot management products can verify the major AI agents against their operators' published IP ranges, reverse DNS or signed requests (see [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/)). Once you know who's asking, you can give each kind of visitor a limit that fits its job: | Visitor | What it's doing | Suggested approach | | --- | --- | --- | | Verified AI assistants (such as ChatGPT-User, Claude-User, Perplexity-User) | Fetching pages because a person asked | Generous limits that allow short bursts. Each request stands in for a customer. | | Verified AI search crawlers (such as OAI-SearchBot) | Indexing pages so you appear in AI answers | A steady crawl rate. Use 429 with Retry-After to pace them, not blocks. | | Verified AI training crawlers (such as GPTBot) | Collecting content for model training | Your business decision. Pace them, or opt out in robots.txt. See [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). | | Unverified traffic claiming an AI agent's name | Unknown, sometimes impersonation | Normal or stricter limits. Never give extra allowance to a name alone. | | Everything else | People, browsers, other bots | Your existing limits, tuned so a busy shopper never hits them. | The right numbers depend on your traffic and infrastructure, so we don't suggest specific figures. Start from what your logs show a normal agent visit looks like, then set limits comfortably above it. ### Limit actions more than pages Abuse mostly targets actions: logins, account sign-ups, discount codes, gift card balances, stock checks and search. Pages that only display information are cheap to serve, especially from a cache. Put your tightest limits on: - Login, sign-up and password reset - Coupon, gift card and checkout submission endpoints - Search and filter endpoints that hit your database - Public APIs and any [MCP server](https://ghostagentlab.com/articles/mcp-webmcp-for-websites/) you offer Keep product, category, pricing, policy and help pages loose, and cache them well. Those are the pages agents need to answer questions about you, and the ones that turn into sales. ### Other ways to reduce load - **Cache aggressively.** A product page served from a CDN cache costs almost nothing, however often it's requested. - **Support conditional requests.** `ETag` and `Last-Modified` headers let crawlers ask "has this changed?" and get a tiny 304 Not Modified response if not. - **Keep your sitemap accurate.** Correct `lastmod` dates help crawlers revisit only what changed. See [XML sitemaps for AI agents](https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/). - **Don't rely on Crawl-delay.** This robots.txt line is ignored by Google and supported unevenly elsewhere. Rate limits with 429 work for every client. ### How to tell if limits are hurting you 1. **Check your logs by agent.** Look at the share of requests from AI assistants that get 429, 403 or challenge responses. A handful of 429s for a crawler is healthy; repeated 429s or 403s for assistant fetchers mean customers' questions are going unanswered. See [how to measure AI agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). 2. **Run [AgentScore](https://ghostagentlab.com/agentscore/).** It requests your home page as a normal browser and as several AI agents, and requests your key pages, cart and checkout as AI assistants. Any error status an agent gets while the browser gets through, including 429, counts as blocked in the bot protection, key pages and checkout checks. 3. **Ask your CDN provider** whether its rate limiting treats verified bots separately, and whether it returns 429 or 403 when a limit is hit. > **A quick checklist.** Rate-limited responses use 429 with Retry-After, never 403 or a CAPTCHA. Verified AI assistants have their own, more generous limits. Names alone earn nothing extra. The strictest limits sit on logins, coupons and search, not on product and policy pages. And someone checks the 429 rate by agent at least monthly. Rate limits and bot protection work together. For the wider picture, read [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/) and, for the business decision about which agents to let in at all, [should you block AI crawlers?](https://ghostagentlab.com/blog/block-or-allow-ai-crawlers/) ### Sources and further reading - [RFC 6585: Additional HTTP Status Codes](https://www.rfc-editor.org/rfc/rfc6585.html) (IETF) - [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) (IETF) - [Reduce the Google crawl rate](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate) (Google) - [RateLimit header fields for HTTP (draft)](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/) (IETF) --- ## Setting up your CDN for AI agents > How to set up your CDN and WAF for AI agents: verified bot categories, signed requests, rule order, challenge versus block, and logging. Published October 9, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/cdn-settings-ai-agents/ ### Key takeaways - Agree which kinds of AI agent you want before changing CDN rules, then build rules on verified bot categories, not names. - A broad block AI bots rule placed above your allow rule for verified agents means the allow rule never runs. - Treat any challenge as a block for agents, and run new rules in log-only mode for a few days before switching them to block. Your CDN and web application firewall (WAF) decide, request by request, who gets to see your site. Most were tuned before AI agents were a real source of customers. This guide walks through the settings that matter, in the order your CDN applies them, so verified AI agents get your pages and the bots you don't want still get stopped. If you're still working out why agents get caught in the first place, start with [Bot protection and CAPTCHAs: stop blocking the agents you want](https://ghostagentlab.com/articles/bot-protection-ai-agents/). This article is the practical follow-on: how to set the CDN up, and how to check it stays that way. ### Start with a decision, not a setting Before anyone changes a rule, agree which kinds of agent you want. Most businesses land somewhere like this: | Kind of agent | What it does | Typical choice | | --- | --- | --- | | AI assistants (for example ChatGPT-User, Claude-User, Perplexity-User) | Fetch a page because a person asked a question about it right now | Allow | | AI search crawlers (for example OAI-SearchBot) | Index pages so they can appear in AI search answers | Allow | | Search engines (Googlebot, Bingbot) | Classic search indexing, which also feeds some AI features | Allow | | AI training crawlers (for example GPTBot, ClaudeBot) | Collect content to train models | A business choice; blocking is common | | Unverified bots claiming any of the names above | Unknown | Challenge, rate limit or block | Write the decision down. It's what your CDN admin implements, and it's what you check against later. Our guides to [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/) and [whether to block or allow AI crawlers](https://ghostagentlab.com/blog/block-or-allow-ai-crawlers/) cover the trade-offs. ### Use verified bot categories, not user agent strings Most large CDNs and bot management products keep a list of verified bots: crawlers and agents whose requests they have confirmed really come from the operator they name. Many also group those bots into categories, so you can write one rule for "AI assistants" instead of maintaining a list of names yourself. Build your rules on that verification, not on the `User-Agent` header. Anyone can send a request that says it's ChatGPT-User. A rule that skips bot checks for that string is an open door for scrapers. Our guide to [telling real AI crawlers from fakes](https://ghostagentlab.com/articles/verify-ai-crawlers/) explains the methods CDNs use behind the scenes: published IP ranges, reverse DNS and signatures. > **Check the category names in your own console.** Vendors name and split these categories differently, and they change them as new agents appear. Confirm which category each agent you care about sits in before you write a rule that depends on it. ### Turn on signature verification where you can IP-based verification struggles with agents that run in the cloud, because their addresses change and are often shared. Web Bot Auth fixes that by having the agent sign each request with a private key, and publish the matching public key on its own domain. It builds on HTTP Message Signatures ([RFC 9421](https://www.rfc-editor.org/rfc/rfc9421)), and is still being standardized, so support varies. If your CDN can verify these signatures, turn it on and treat a valid signature as verification. If it can't yet, nothing breaks: signed requests are ordinary requests with extra headers. Our post [Why we sign every request our agents send](https://ghostagentlab.com/blog/signed-requests/) explains how it works in practice. ### Get the rule order right CDNs apply rules in a sequence, and the first matching action often wins. Most agent problems we see come from order, not from a missing rule. A sensible order looks like this: 1. **Security rules for everyone.** Managed WAF rules that catch attacks such as SQL injection should apply to every request, verified bots included. Verification proves who sent a request, not that the request is safe. 2. **Hard blocks you need for legal or business reasons.** For example, blocked training crawlers, or country restrictions you're required to enforce. See [Geo-blocking, VPN blocks and AI agents](https://ghostagentlab.com/articles/geo-blocking-ai-agents/) for the country case. 3. **Allow verified agents you want**, by category or verified identity. "Allow" here should mean "skip bot challenges and bot scoring", not "skip all security". 4. **Rate limits** for everyone, including verified agents, set high enough for normal use. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). 5. **Bot scoring and challenges** for whatever is left: unverified automation, impostors and suspicious traffic. The common mistake is a broad "block AI bots" or "challenge automated traffic" rule placed above the verified-agent allow rule. The allow rule then never runs. ### Challenge or block: choose deliberately Most bot tools offer several actions. For a person they feel very different. For an AI agent, most of them end the same way. | Action | What a person sees | What an AI agent gets | | --- | --- | --- | | Block | An error page | An error page; it tells the user it can't reach you | | JavaScript or "managed" challenge | A short wait, usually invisible | Usually a dead end: most assistant fetchers don't run scripts | | Interactive challenge or CAPTCHA | A puzzle or checkbox | A dead end: agents can't, and shouldn't, solve them | | Rate limit (429 with Retry-After) | Rarely seen | A clear signal to slow down and try again | | Log only | Nothing | The page | So treat a challenge as a block when you're thinking about agents. Keep challenges for traffic you genuinely don't want, and for high-risk actions such as login, account creation and payment. Product, pricing, policy and help pages should open for verified agents without one. ### Log enough to see what happened When an assistant says it "couldn't access" your site, someone has to find out why. Make sure your CDN logs, or the samples it keeps, include for each request: - The user agent, and whether the CDN treated the request as a verified bot (and in which category) - The action taken (allowed, challenged, blocked, rate limited) and the ID of the rule that took it - The response status code and the path - The client country and network, if you use location rules Before you switch a new rule to block, run it in log-only (sometimes called "simulate" or "count") mode for a few days and check what it would have caught. Then review blocked and challenged agent traffic regularly. Our guides to [measuring AI agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/) and [finding the errors AI agents hit in your logs](https://ghostagentlab.com/articles/agent-errors-in-logs/) show what to look for. ### Notes for common vendors Products change quickly and menus move, so treat these as pointers, and check your vendor's current documentation for the exact names and the plan you're on. - **Cloudflare** maintains a verified bots program and documents bot categories that you can reference in custom rules, including separate groupings for AI crawlers and other AI agents. It also offers settings aimed at blocking AI crawlers; check exactly which agents those cover before turning them on. Cloudflare has published work on Web Bot Auth, so check whether signature verification is available on your plan. - **Akamai** bot management sorts known bots into categories and lets you set an action for each. Find where the AI-related categories sit and which action they currently get. - **Fastly** offers bot management and WAF features with its own way of identifying known bots. Check how verified bots are flagged in its rules, and where that check runs relative to your other rules. - **AWS WAF** has a bot control managed rule group that labels requests, including whether a bot is verified. Rules later in the web ACL can act on those labels, so order and labels together decide what agents get. If you use a dedicated bot product in front of or behind your CDN, the same questions apply to it. Two layers mean two places an agent can be stopped. ### Check it from the outside A configuration that looks right in the console can still behave differently in practice. Test it as an agent would: > 1. Run [AgentScore](https://ghostagentlab.com/agentscore/). It requests your home page as a normal browser and as AI agents including ChatGPT-User, Claude-User, Perplexity-User and OAI-SearchBot, and fails the bot protection check if an agent is blocked or challenged while the browser gets through. Other checks test your product, pricing and cart pages the same way, and look for CAPTCHAs on arrival and at checkout. > 2. Look at your CDN's logs for the scan and confirm which rule acted on each request. > 3. Re-run the scan after every bot or WAF change, and after your vendor announces changes to its bot categories. One caution: AgentScore's requests under those agent names aren't signed and don't come from those operators' networks, so a correctly configured CDN that only allows *verified* agents may still challenge them. That's the right behavior. Read a failure alongside your logs: if real, verified agent traffic is getting through, your setup is working. Our post on [why our scanner wears other names](https://ghostagentlab.com/blog/scanner-wears-other-names/) explains the trade-off. ### Sources and further reading - [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html) (IETF) - [HTTP Message Signatures for automated traffic Architecture (Web Bot Auth draft)](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/) (IETF) - [Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) (Cloudflare) - [AWS WAF Bot Control](https://docs.aws.amazon.com/waf/latest/developerguide/waf-bot-control.html) (AWS) --- ## Geo-blocking, VPN blocks and AI agents > Why country, VPN and data center blocks and IP-based redirects catch AI agents visiting from cloud servers abroad, and how to let verified agents through. Published October 9, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/geo-blocking-ai-agents/ ### Key takeaways - AI assistants visit from cloud data centers, sometimes in another country, so location and network rules can block them or show the wrong store. - Redirecting visitors by location can send an agent to another country's site, where it confidently quotes the wrong prices, stock and delivery times. - Start by listing every location and network rule, keep the legally required ones, and let verified AI agents past the rest. A shopper in Manchester asks an AI assistant about your store. The assistant doesn't visit from Manchester. It visits from a cloud data center, quite possibly in another country. If your site blocks, challenges or redirects visitors by location or network, that one fact can decide whether the assistant sees your real store, the wrong store, or nothing at all. ### Where AI agents actually come from When a person asks an assistant a question, the page request doesn't come from their phone or laptop. It comes from the assistant's servers. The same is true for AI search crawlers and for browser agents that run in the cloud. That has three consequences: - **The country is the server's, not the shopper's.** Agent traffic is concentrated in the regions where AI companies run their infrastructure. Those are rarely the same as your customers' locations. - **The network is a data center.** Agent requests come from cloud and hosting providers, the same networks that a lot of abusive automation uses. Reputation lists often score them as high risk. - **Addresses change.** Agents that run on shared cloud infrastructure don't always come from fixed, published IP ranges. None of this is a sign of bad intent. It's simply how AI agents work. But it means rules written with human visitors in mind can treat agents very differently from the people they're acting for. ### Four rules that catch agents #### Country blocks Some sites block whole countries, often to cut fraud or because they don't ship there. If an assistant's servers sit in a blocked country, every request it makes for every shopper is turned away, including shoppers in countries you serve. #### VPN, proxy and data center blocks Blocking "anonymizing" networks or hosting providers is a common anti-fraud and anti-scraping setting. It catches nearly all AI agent traffic, because nearly all of it comes from hosting providers. It's one of the most common reasons an assistant can reach a competitor's site but not yours. #### Automatic geo-redirects Many international stores redirect visitors to a country site based on IP address. An agent fetching your UK product page for a UK shopper may be sent to your US site instead, then quote US prices, US stock and US delivery times. The answer looks confident and is wrong. Some redirects go to a country picker page, which leaves the agent with no product at all. #### Location-based content Some sites quietly change prices, currency, product ranges or policies based on the visitor's location, without changing the URL. An agent then describes a version of your store the shopper will never see. ### What to do instead The aim is to keep the protection you need while letting verified agents see the right content. #### 1. Separate legal requirements from preferences Some location rules are required: sanctions, licensing, or content you're not allowed to show in certain places. Keep those, and get advice on how they apply to automated visitors. Many others are preferences, such as "we don't ship there, so block it". For those, ask whether blocking the server's location actually achieves anything. You already check the delivery address at checkout, which is where a shipping restriction belongs. #### 2. Let verified agents past location and network rules Most CDNs and bot management tools can tell verified bots apart from other traffic. Add an exception so that verified AI assistants and AI search agents skip the country, VPN and data center rules that are about fraud or scraping. Put it in the right place in your rule order, and never base it on the user agent alone. Our guide to [setting up your CDN for AI agents](https://ghostagentlab.com/articles/cdn-settings-ai-agents/) covers rule order, and [telling real AI crawlers from fakes](https://ghostagentlab.com/articles/verify-ai-crawlers/) covers verification. > **Don't allow whole cloud providers.** Lifting a data center block for an entire cloud network lets in every scraper hosted there. Allow verified agents, and keep the block for everything else. #### 3. Let the URL decide the country, not the IP address Give each country or language its own URL, such as `northwind.example/en-gb/` or `uk.northwind.example`, and serve the content that URL promises, whoever asks. If you want to steer people to their local site, show a banner suggesting it instead of redirecting. Agents and people can then follow the link that matches the shopper. Mark the alternatives with `hreflang` so agents and search engines can find them: ``` ``` Our guide to [languages, currencies and regions](https://ghostagentlab.com/articles/international-sites-ai-agents/) goes further on international setups. #### 4. Make the default page useful Some agents will still land on your default site. Make sure it's a real store, not a country picker. State which country and currency the prices are in, and link clearly to the other country sites. Put the same information in your structured data, so an agent knows the price it read is in US dollars and not pounds. See [machine-readable prices](https://ghostagentlab.com/articles/machine-readable-prices/). #### 5. Use rate limits for the rest For traffic you can't verify, from data centers or anywhere else, rate limits are usually a better tool than outright blocks. They stop bulk scraping while still letting through a small number of requests made on someone's behalf. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). ### How to find out if this affects you | Look for | Where | What it suggests | | --- | --- | --- | | Country, ASN, "hosting" or "anonymous proxy" rules | CDN, WAF or bot management settings | Agents may be blocked or challenged by location or network | | Redirects based on IP country | CDN rules, e-commerce platform settings, geolocation apps or plugins | Agents may see the wrong country's store | | 403s, challenges or 302 redirects for ChatGPT-User, Claude-User or Perplexity-User | CDN or server logs | Real agent visits are being turned away or sent elsewhere | | Assistants quoting the wrong currency or prices | Ask a few assistants about your products | A redirect or location-based content is reaching agents | An [AgentScore](https://ghostagentlab.com/agentscore/) scan helps too. Its requests come from cloud servers, like real agents' requests do, and it compares what AI agents get from your home page, product, pricing and cart pages with what a normal browser gets from the same servers. It can't show what visitors in every country see, so a block that only applies to certain countries may not show up. Pair it with a look at your rules and logs. Our guide to [finding the errors AI agents hit in your logs](https://ghostagentlab.com/articles/agent-errors-in-logs/) shows how to pull out agent requests and their status codes. ### A short checklist > 1. List every location and network rule on your CDN, WAF, bot tool and e-commerce platform, and note which are legally required. > 2. Add an exception for verified AI assistants and AI search agents to the rules that aren't. > 3. Replace automatic IP redirects with country URLs, `hreflang` and a suggestion banner. > 4. Make sure your default site shows real products, with the currency and country stated. > 5. Check your logs for agent requests that get 403s, challenges or redirects, and re-run [AgentScore](https://ghostagentlab.com/agentscore/) after each change. ### Sources and further reading - [Localized versions of your pages (hreflang)](https://developers.google.com/search/docs/specialty/international/localized-versions) (Google Search Central) - [Managing multi-regional and multilingual sites](https://developers.google.com/search/docs/specialty/international/managing-multi-regional-sites) (Google Search Central) --- ## Login walls and gated content: what AI agents can reach > What AI agents can't reach behind logins and paywalls, what to keep public, how to hand off to the customer, and paywall structured data for publishers. Published October 9, 2026 by Ghost Agent Labs · Access · https://ghostagentlab.com/articles/login-walls-ai-agents/ ### Key takeaways - Anything behind a login or paywall is, for most AI agents, simply not there, so they can only use what's public. - Hiding prices, product details or policies behind sign-in means an assistant says it doesn't know or quotes a competitor who shows them. - Keep the wall in front of private and paid content only, and offer guest checkout so agents can finish a purchase. An AI agent acting for a customer usually doesn't have that customer's password, and it can't create accounts or pay for subscriptions on a whim. Anything you put behind a login or a paywall is, for most agents, simply not there. That's often the right call. But many sites hide things that should be public, and lose recommendations because of it. ### What agents can and can't reach Different kinds of agent meet a login wall in different ways: | Kind of agent | Behind a login or paywall | | --- | --- | | AI assistants fetching a page for a question | Get the login page or the paywall, and answer from whatever is visible | | AI search and training crawlers | Index only what's public, unless you choose to show them more | | Browser agents carrying out a task | Usually stop and ask the person to sign in, or give up. Some can hand control back to the person for the sign-in step | So the question for each page is simple: if an agent can only see the public version, can it still answer the customer's question and move them forward? ### What to keep public These are the things agents are most often asked about. If they sit behind a login, the agent either says it doesn't know or quotes a competitor who shows them openly. - **Prices.** "Sign in to see price" and "add to cart to see price" hide the one fact shoppers ask about most. If you can't show an exact price, show a starting price or a range. See [machine-readable prices](https://ghostagentlab.com/articles/machine-readable-prices/). - **Product information.** Specifications, sizes, materials, compatibility and stock. A catalog that only appears after sign-in is invisible to agents. - **Policies.** Shipping, returns, warranty and payment options. These decide many purchases, and agents quote them directly. See [shipping, returns and FAQ pages AI assistants can quote](https://ghostagentlab.com/articles/policy-pages-ai-assistants/). - **Plans and pricing for software.** What each plan includes and costs, and whether there's a free trial. See [agent readiness for software companies](https://ghostagentlab.com/blog/agent-readiness-for-saas/). - **Help content.** Setup guides, FAQs and documentation. If your help center needs a login, assistants can't answer support questions about your product. - **Contact and location details.** Opening hours, addresses and how to reach you. Trade and wholesale businesses often hide everything, because prices vary by customer. Consider showing the catalog and list prices publicly, with a note that account holders get their own prices after signing in. The agent can then recommend you and send the buyer to log in for the final number. ### What can stay behind the wall Plenty of content belongs there: order history, saved addresses, account settings, customer-specific prices, members-only content and paid articles. The goal isn't to remove logins. It's to make sure the wall sits in front of private and paid things, not in front of the information a customer needs to decide. ### Handing off to the person When an agent does reach something that needs the customer, make the handoff smooth: - **Say what's needed in plain text.** "Sign in to see your order history" is something an agent can relay. A login form with no explanation isn't. - **Return to the same place.** After sign-in, send the person back to the page or cart they were on, with the cart intact, so the agent's work isn't lost. - **Don't require an account to buy.** Guest checkout lets an agent finish a purchase without creating an account for someone. AgentScore checks for it in *Shoppers can check out without an account*, and our guide to [guest checkout](https://ghostagentlab.com/articles/guest-checkout-ai-agents/) explains why it matters. - **Use proper status codes.** Redirect to a login page, or return 401 or 403, rather than a 200 page that looks like content but isn't. Agents and crawlers can then tell "this needs a login" from "this page says nothing". - **Make sign-in forms agent-friendly.** Labelled fields and standard `autocomplete` values help browser agents and password managers alike. See [sign-up, booking and lead forms AI agents can finish](https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/). ### Paywalls and structured data for publishers Publishers face a specific version of this. You may want search engines to see the full article so they can rank it, while readers see a paywall. Showing crawlers different content from readers can look like cloaking. Google's documented answer is paywalled content structured data: you mark the article as not free, and point to the part that's behind the paywall. ``` ``` The `cssSelector` must match a class on the element that wraps the paid content. Google explains the details in its [paywalled content guidelines](https://developers.google.com/search/docs/appearance/structured-data/paywalled-content), and the properties are defined at [schema.org](https://schema.org/isAccessibleForFree). Two things to keep in mind: - **The markup describes access; it doesn't grant it.** Whether you show full articles to any crawler, including AI search and training crawlers, is a business and licensing decision. Make it deliberately, per agent, and reflect it in [robots.txt](https://ghostagentlab.com/articles/robots-txt-ai-agents/) and your CDN rules. - **Give agents something to cite.** A clear headline, a summary paragraph and the key facts above the paywall let assistants describe and link to your article even when they can't read it all. A paywall that hides everything, headline included, gets you nothing. ### How to check your own site > 1. Open a private browser window, signed out, and try to find a product's price, your returns policy and your contact details. Anything you can't find, an agent can't either. > 2. Add an item to the cart and go to checkout. If you're asked to sign in or create an account with no guest option, agents buying for someone will stop there. > 3. Ask a few AI assistants what your products cost and what your returns policy is. Wrong or missing answers often point to content behind a wall. > 4. Run [AgentScore](https://ghostagentlab.com/agentscore/) for the guest checkout, key pages and price checks, and use [Ghost Agents](https://ghostagentlab.com/ghost-agent/) to run a real journey and see exactly where an agent gets stuck. ### Sources and further reading - [Subscription and paywalled content markup](https://developers.google.com/search/docs/appearance/structured-data/paywalled-content) (Google Search Central) - [isAccessibleForFree (schema.org property)](https://schema.org/isAccessibleForFree) (schema.org) - [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) (IETF) - [HTML Standard: Autofill](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#autofill) (WHATWG) --- ## Product data that AI shopping agents can read > How to describe products, prices, stock, shipping and returns with schema.org structured data, with a complete JSON-LD example and the rules that keep it trustworthy. Published October 8, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/ ### Key takeaways - Structured data gives AI agents your product's name, price, currency, stock and return terms in a standard format, so they don't guess. - Markup that disagrees with the visible page is worse than none, because agents may quote the wrong price and search engines may ignore it. - Check what your platform already outputs, then fill common gaps such as stock status, product codes, and shipping and return details. When an AI agent compares products for someone, it needs facts it can trust: the exact name, the price, the currency, whether it's in stock, how shipping and returns work. Structured data hands it those facts in a standard format, instead of making it guess from your page layout. ### What structured data is Structured data is a small block of machine-readable information in your page's HTML that describes what the page is about, using the shared vocabulary at [schema.org](https://schema.org/). The most common format is JSON-LD: a ` ``` ### The fields that matter most | Field | Why agents need it | | --- | --- | | `name` | The exact product, including size or pack where it matters | | `offers.price` and `priceCurrency` | The price the customer will actually pay, as a plain number, with no currency symbol | | `offers.availability` | Whether it can be bought now. Agents skip products they think are out of stock | | `sku`, `gtin`, `mpn` | Let an agent match your product to the same item elsewhere when comparing | | `brand` | Answers "do you have anything from …?" questions | | `aggregateRating` | Lets agents weigh quality, but only include it if the reviews are on the page | | `hasMerchantReturnPolicy`, `shippingDetails` | Answer the questions people ask before buying, without a trip to your policy pages | ### Products with options If a product comes in sizes or colors with different prices or stock, describe each variant. Schema.org's `ProductGroup` type groups variants under one parent, with `variesBy` naming what changes (such as size or color) and `hasVariant` listing each `Product` with its own offer. At minimum, make sure the price and availability on the page match the variant a visitor lands on. ### Beyond product pages - **Home page:** `Organization` (or `LocalBusiness` with address and opening hours) with your name, logo, website, contact details and social profiles. AgentScore looks for structured data on your home page. - **Category pages:** `BreadcrumbList` shows where a page sits in your catalog, and helps agents move between categories. - **Help pages:** `FAQPage` for genuine question-and-answer content, such as shipping and returns questions. - **Software pricing:** `SoftwareApplication` or `Product` with an `Offer` for each plan. ### Rules that keep it trustworthy - **Match the page.** Every value must match what a person sees. A price in the markup that differs from the page is worse than none: an agent may quote the wrong one, and search engines may ignore your markup. - **Keep it current.** Generate it from the same data as the page, so prices and stock update together. Hand-written markup goes stale. - **Put it in the HTML.** Markup added by JavaScript after the page loads is invisible to agents that don't run JavaScript. AgentScore flags this as a warning. See [why AI agents can't see JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). - **Prefer JSON-LD.** Microdata and RDFa work, but JSON-LD keeps the data in one tidy block that's easier to read and maintain. - **Don't mark up what isn't there.** No ratings without visible reviews, no FAQ markup for questions that aren't on the page. ### How to add it 1. **Check what you already have.** Many e-commerce platforms and themes output basic product markup. View the page source and search for `application/ld+json`. 2. **Fill the gaps.** Common missing pieces are `availability`, identifiers like GTIN, and return and shipping details. Platform apps and SEO plugins can add these, or your developer can extend the theme template. 3. **Validate.** Paste the page URL into the [Schema Markup Validator](https://validator.schema.org/) to check the markup is valid, and Google's Rich Results Test to see which fields Google can use. 4. **Run [AgentScore](https://ghostagentlab.com/agentscore/).** The structured data check passes when your home page has JSON-LD in its HTML, and warns if it only appears after JavaScript runs. Structured data is one part of being readable to agents. The other big one is making sure your content is in the HTML at all; see the guide to [JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/), or the full [agent readiness guide](https://ghostagentlab.com/articles/agent-readiness/). ### Sources and further reading - [Product (schema.org type)](https://schema.org/Product) (schema.org) - [ProductGroup (schema.org type)](https://schema.org/ProductGroup) (schema.org) - [Intro to product structured data on Google](https://developers.google.com/search/docs/appearance/structured-data/product) (Google Search Central) - [General structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) (Google Search Central) --- ## Why AI agents can't see your JavaScript-only content > Many AI agents read only the HTML your server sends. How to check what they see, what usually goes missing, and how to fix it with server rendering. Published October 8, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/javascript-content-ai-agents/ ### Key takeaways - Many AI agents read only the HTML your server sends, so content that appears only after scripts run may be invisible to them. - Ranking well in Google can hide the problem, because Google runs JavaScript while AI assistants may see a nearly blank page. - Ask your developers to render the essentials on the server first: product name, price, availability, key specifications, shipping, returns and plan pricing. Open a modern website with JavaScript turned off and you'll often see a logo, a spinner and not much else. That's what many AI agents see too. If your prices, products or policies only appear after scripts run, a large share of AI traffic never sees them. ### How agents read a page When a browser loads a page, it does two things: it downloads the HTML your server sends, then runs the JavaScript, which may fetch more data and build much of what you see. Agents differ in how far they go: | Agent | Typically reads | | --- | --- | | Training and AI search crawlers | The HTML only. Running JavaScript for billions of pages is slow and expensive, so most AI crawlers don't. | | Assistant fetchers (fetching a page because someone asked) | Usually the HTML only, and they have a few seconds at most. | | Browser agents | The full page, including JavaScript, but they're slow and impatient. Long loading states and content that appears late still trip them up. | Googlebot does run JavaScript, which is why many sites that rely on it rank fine in Google search. That has hidden the problem: the same site can be well indexed by Google and nearly blank to AI assistants. ### How to check your own site 1. **View the source.** In your browser, use View Page Source (not the developer tools inspector, which shows the page after JavaScript). Search for a product name or price you can see on the page. If it isn't in the source, it's added by JavaScript. 2. **Turn JavaScript off.** In Chrome's developer tools, open the command menu, type "Disable JavaScript", and reload. What's left is roughly what an HTML-only agent sees. 3. **Fetch it from the command line.** `curl -s https://yourdomain.com/your-page | grep -i "your price"` shows exactly what your server sends. 4. **Run [AgentScore](https://ghostagentlab.com/agentscore/).** It compares the text in your HTML with the text after JavaScript runs. It passes when at least 70% of the text is already in the HTML, and warns or fails below that. ### What usually goes missing - **Prices and stock** loaded from an API after the page appears, so the HTML says "Loading…" or nothing at all. - **Whole pages in single-page apps**, where the HTML is an empty `
` and everything else is built in the browser. - **Product lists with infinite scroll** that load more items only as a person scrolls, with no paginated links to follow. - **Content inside tabs and accordions** fetched only when clicked, such as specifications, sizing and shipping details. - **Structured data** injected by a tag manager or script, which agents reading the HTML never see. See [product data AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). - **Reviews** from third-party widgets that load in the browser. ### How to fix it The aim is simple: the important content should be in the HTML your server sends. JavaScript can still make the page interactive afterwards. #### Server-side rendering (SSR) The server builds the full HTML for each request, then JavaScript takes over in the browser. Modern frameworks support this directly: Next.js and Remix for React, Nuxt for Vue, SvelteKit for Svelte, and Angular's server rendering. If your site is a single-page app built with one of these, turning on server rendering for key pages is usually the biggest single fix. #### Static generation Pages are built to HTML ahead of time, at deploy. It's ideal for content that changes rarely, such as marketing pages, docs, policies and blog posts. Astro, Eleventy, Hugo and the frameworks above can all do it, often alongside server rendering for pages that change often. #### Render the essentials, enhance the rest You don't need to render everything on the server. Prioritize what agents ask about: product name, price, availability, key specifications, shipping and returns, and plan pricing. Personalized recommendations, recently viewed items and chat widgets can stay in JavaScript. #### Pre-rendering as a stopgap Pre-rendering services run your JavaScript ahead of time and serve the finished HTML to bots. It can work as a short-term fix, but it means maintaining two versions of each page, and it depends on correctly recognizing every agent. Google describes serving bots differently ("dynamic rendering") as a workaround rather than a long-term solution. Server rendering is better. ### Smaller fixes that help - Replace infinite scroll with, or add, real paginated links (`?page=2`) agents can follow. - Put tab and accordion content in the HTML and hide it with CSS, instead of fetching it on click. - Move JSON-LD from your tag manager into the page template. - Include a summary of reviews (average rating and count) in the HTML, even if the full reviews load later. - Give the page a meaningful `` and `<h1>` in the HTML, never only after JavaScript runs. Once your content is in the HTML, the next question is whether agents can find their way around it and use it. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). ### Sources and further reading - [Understand JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) (Google Search Central) - [Dynamic rendering as a workaround](https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering) (Google Search Central) - [Disable JavaScript (Chrome DevTools)](https://developer.chrome.com/docs/devtools/javascript/disable) (Chrome for Developers) --- ## Alt text for AI agents: describing product images > Why AI agents rely on alt text, what AgentScore checks, and how to write useful descriptions for product photos across a whole catalog. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/image-alt-text-ai-agents/ ### Key takeaways - Most AI agents read the alt text attached to product photos rather than the photos, so missing descriptions leave images effectively invisible. - File names, keyword lists and identical labels on every shot tell agents little, and AI-written alt text can describe the wrong color. - Fix the product image template to fall back to the product name and variant, then hand-write descriptions for your bestsellers. Your product photos do a lot of selling. They show the color, the cut, the size in someone's hand. Most AI agents never see them. What they read instead is the alt text: the short description attached to each image in your HTML. If it's missing, the photo might as well not be there. ### Why images are a blind spot for agents An AI agent helping someone shop usually reads your page as text. It fetches the HTML, strips out the layout and works from what's left. Images arrive as file names and URLs, which tell it very little. `IMG_4471_final.jpg` doesn't say whether the jacket is navy or black. Some agents do look at the page as a picture, taking screenshots and reading them with a vision model. They can make out a photo, but slowly, at a cost, and not always correctly. Text is faster, cheaper and unambiguous, so even those agents lean on it when it's there. Alt text is the bridge. It's an attribute on the image tag, written for anyone who can't see the image: screen reader users, people on slow connections, search engines and now AI agents. Writing it well is the same work for all of them. We make that case in [Accessibility work is agent readiness work](https://ghostagentlab.com/blog/accessibility-is-agent-readiness/). ### What AgentScore checks The "Images have text descriptions" check is part of the Readability category. It loads your home page in a real browser and looks at the images people can actually see, ignoring tiny ones under 32 pixels wide such as icons and tracking pixels. For each one it asks a simple question: does it have an `alt` attribute at all? | Result | What it means | | --- | --- | | Pass | 90% or more of visible images have alt text, or there are no content images on the page | | Warning | Between 60% and 90% have alt text | | Fail | Fewer than 60% have alt text | An empty `alt=""` counts as described. That's deliberate: it's the correct way to mark an image as decorative, and it tells agents and screen readers to skip it. The check can't judge whether your descriptions are any good, so that part is up to you. It's a small check by weight, because a missing description rarely stops an agent on its own. But the same gaps tend to show up elsewhere. A linked product image with no alt text is also a link with no name, and that's caught by the "Buttons and links have names" check in Navigability, which carries more weight. ### How to write alt text for product images Describe what a shopper needs to know from the photo, in the words they'd use. Lead with the product, then add the details the image actually shows. | Weak | Better | | --- | --- | | `alt="product"` | `alt="Northwind waxed cotton field jacket in navy, front view"` | | `alt="IMG_4471"` | `alt="Field jacket in navy, close-up of brass zip and corduroy collar"` | | `alt="best jacket waterproof jacket mens jacket sale"` | `alt="Model, 6 ft 1 in, wearing the field jacket in size L"` | | No alt attribute | `alt=""` if the image is purely decorative | - **Name the product and the variant.** If the photo shows the navy version, say navy. Agents use this to match images to options. - **Say what's different about each shot.** Front, back, detail, in use, on a model. Five images all called "Field jacket" waste the opportunity. - **Include facts the photo proves.** Model height and size worn, what's in the box, the scale of an object next to something familiar. - **Keep it short.** One short sentence is usually enough. Long specifications belong in the product description, not the image. - **Don't stuff keywords.** Agents read alt text as a description. A list of search terms reads as noise and helps nobody. - **Don't start with "image of".** Everyone reading it already knows it's an image. ### Decorative images, logos and linked images Not every image needs words. Background textures, divider graphics and lifestyle photos that repeat the headline next to them can take `alt=""`. That's different from leaving `alt` out: an empty value says "nothing to see here", while a missing one leaves agents and screen readers guessing, and some will read out the file name instead. Images that are links need extra care, because the alt text becomes the link's name. A product tile where the whole image links to the product page should have alt text that works as a link, such as the product name. Your logo in the header links home, so `alt="Northwind home"` or simply your brand name works better than `alt="logo"`. Text baked into images is a common trap on home pages. A banner that says "40% off outerwear this weekend" as pixels is invisible to most agents. Put the offer in real HTML text over the image, or at minimum repeat it in the alt text. ### Doing this at catalog scale A store with thousands of products can't write every description by hand. You don't need to. Most of it can be generated from data you already have. 1. **Fix the template first.** Make sure your theme outputs an `alt` attribute on every product image, and falls back to the product name and variant when no custom text is set. That alone gets most stores to a pass. 2. **Use the image metadata your platform stores.** Shopify, WooCommerce, Magento and most other platforms have an alt text field per image. Fill it on new products as part of the listing checklist. 3. **Backfill your bestsellers.** Write proper descriptions for the products that earn the most, where shoppers are most likely to ask an assistant about them. 4. **Review generated text before publishing.** Tools that write alt text with AI are useful for a first draft, but they can describe the wrong color or invent details. Check a sample, especially for variants. 5. **Add it to the content workflow.** Merchandisers and content teams usually own this, not developers. Make alt text a required field when new images are uploaded. > **Check it yourself in a minute:** open a product page, right-click a photo and choose Inspect. Look for `alt="…"` on the `<img>` tag. If it's missing, or it just says "image", agents are getting nothing. ### Where alt text fits Alt text is the visual part of a bigger job: making sure everything a shopper learns from your page is also available as text. The other parts are prices and stock in a form agents can read (see [Prices AI agents can read](https://ghostagentlab.com/articles/machine-readable-prices/) and [Product data that AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/)), and content that's in the HTML rather than added by JavaScript (see [our guide to JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/)). Run [AgentScore](https://ghostagentlab.com/agentscore/) to see how your home page does, then spot-check a few product pages by hand. For the full picture of what agents need from a site, start with the [agent readiness guide](https://ghostagentlab.com/articles/agent-readiness/). ### Sources and further reading - [HTML Standard: Requirements for providing text to act as an alternative for images](https://html.spec.whatwg.org/multipage/images.html#alt) (WHATWG) - [Decorative images](https://www.w3.org/WAI/tutorials/images/decorative/) (W3C Web Accessibility Initiative) - [Functional images](https://www.w3.org/WAI/tutorials/images/functional/) (W3C Web Accessibility Initiative) --- ## XML sitemaps for AI agents > How AI agents and crawlers use your sitemap, what AgentScore checks, what to include and how to keep it accurate. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/ ### Key takeaways - A sitemap lists the pages you want found, helping AI agents and crawlers reach products and policies your navigation might hide. - Hand-made sitemaps go stale within weeks, and some site builders serve an ordinary web page at the sitemap address instead. - Open your robots.txt, check for a Sitemap line, and confirm the file it points to loads as a list of your pages. A sitemap is a list of the pages on your site that you want found, in a format machines read without effort. Search engines have used them for about twenty years. AI agents and AI search crawlers use them too, to find pages your navigation hides and to discover new products quickly. It's one of the cheapest readiness wins there is. ### What a sitemap does for AI agents An agent arriving at your site has two ways to find things: follow links from the home page, or read a list someone has prepared. Following links is slow and fragile. Menus built with JavaScript, products buried four clicks deep and pages that only appear in search results can all be missed. A sitemap solves that. It's an XML file, usually at `/sitemap.xml`, that lists your important URLs. Crawlers from AI search engines use it to decide what to fetch. Agents doing a task can use it to locate a product or policy page directly. And it gives every system the same, complete picture of what you publish. It doesn't replace good links. A page that's only in your sitemap is still a page most visitors and many agents never reach. Treat the sitemap as the safety net under your navigation, not instead of it. Our guide to [linking key pages from the home page](https://ghostagentlab.com/articles/link-key-pages-home-page/) covers the other half. ### What AgentScore checks The "Valid sitemap" check, in the Readability category, looks for your sitemap the way crawlers do: 1. It reads `Sitemap:` lines in your [robots.txt](https://ghostagentlab.com/articles/robots-txt-ai-agents/). 2. It then tries `/sitemap.xml` and `/sitemap_index.xml`. 3. For each, it checks the file loads with a 200 status and is a real sitemap: XML with a `<urlset>` or `<sitemapindex>` element. It passes when it finds one, and reports how many entries it holds and whether it's listed in robots.txt. It fails if none of those places has a valid sitemap. A home page or a "not found" page served at `/sitemap.xml` doesn't count, which catches a common problem with some site builders. The sitemap also feeds other checks. AgentScore uses it to find a product or pricing page when the home page doesn't link to one. If the sitemap is an index, it reads product sitemaps first, since most platforms name them that way. That's why a missing sitemap can leave other checks marked "not tested": without a product page to look at, there's nothing to check prices or product data on. ### What a good sitemap looks like The format is defined at [sitemaps.org](https://www.sitemaps.org/protocol.html). A minimal one for a small store: ``` <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://northwind.example/</loc> <lastmod>2026-10-01</lastmod> </url> <url> <loc>https://northwind.example/products/ethiopia-yirgacheffe</loc> <lastmod>2026-10-07</lastmod> </url> <url> <loc>https://northwind.example/pages/shipping</loc> <lastmod>2026-08-14</lastmod> </url> </urlset> ``` Larger sites split it up. Each sitemap file can hold up to 50,000 URLs, and a sitemap index lists the individual files: ``` <?xml version="1.0" encoding="UTF-8"?> <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap><loc>https://northwind.example/sitemap_products_1.xml</loc></sitemap> <sitemap><loc>https://northwind.example/sitemap_collections_1.xml</loc></sitemap> <sitemap><loc>https://northwind.example/sitemap_pages_1.xml</loc></sitemap> </sitemapindex> ``` Then tell crawlers where it is, with one line in robots.txt: ``` Sitemap: https://northwind.example/sitemap.xml ``` ### What to include, and what to leave out | Include | Leave out | | --- | --- | | Product and category pages you want people to find | Cart, checkout and account pages | | Pricing, plans and sign-up pages | Internal search results and filtered URLs | | Shipping, returns, FAQ and contact pages | Pages that redirect elsewhere | | Guides, articles and other evergreen content | Pages marked noindex, or blocked in robots.txt | | The canonical URL of each page | Duplicates with tracking parameters or session IDs | The rule of thumb: list the pages you'd be happy for an AI assistant to send a customer to, each at its one true address. If you include a page in the sitemap but block it in robots.txt, you're sending crawlers mixed signals. ### Keep it accurate - **Generate it automatically.** Shopify, WooCommerce, Magento, BigCommerce, WordPress SEO plugins and most modern frameworks produce a sitemap for you. A hand-made sitemap goes stale the week after it's written. - **Make `lastmod` honest.** It should change when the content does, such as a price or stock change, not every time the file is rebuilt. [Google's documentation](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) says it uses `lastmod` only when it's consistently accurate, and other crawlers are likely to treat it similarly. - **Don't bother with priority and changefreq.** They're in the protocol, but Google says it ignores them. Your time is better spent on accurate dates. - **Remove dead pages.** Discontinued products that return 404, or redirect to a category, shouldn't stay listed. - **Keep it reachable.** Make sure bot protection doesn't challenge crawlers asking for the sitemap. See [our guide to bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/). ### Sitemap, llms.txt or both? Both, because they do different jobs. A sitemap is complete: every page you want found, with no explanation. An [llms.txt file](https://ghostagentlab.com/articles/llms-txt/) is selective: a short, described guide to the pages that matter most, written for AI. A crawler indexing your catalog wants the sitemap. An assistant trying to answer "what's their returns window?" benefits from llms.txt pointing straight at the returns page. > **Quick check:** open `yourdomain.com/robots.txt` and look for a `Sitemap:` line. Then open the URL it gives. You should see XML listing your pages, not a web page. If either step fails, that's your first fix. ### Next steps 1. Confirm your platform generates a sitemap, and that it loads. 2. Add a `Sitemap:` line to robots.txt if it's missing. 3. Spot-check that your bestselling products and your policy pages are listed. 4. Run [AgentScore](https://ghostagentlab.com/agentscore/). The "Valid sitemap" check will show what it found and how many entries it counted. For how the sitemap fits with the other things agents need, see the [agent readiness guide](https://ghostagentlab.com/articles/agent-readiness/). ### Sources and further reading - [Sitemaps XML format (protocol)](https://www.sitemaps.org/protocol.html) (sitemaps.org) - [What is a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) (Google Search Central) - [Build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) (Google Search Central) --- ## Titles, descriptions and headings AI agents understand > How AI agents use page titles, meta descriptions, headings and the lang attribute, and how to write each so agents pick the right page. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/titles-descriptions-headings-ai/ ### Key takeaways - AI agents rely on page titles, descriptions and headings to decide whether a page answers their question and where to look. - Vague or duplicated labels such as "Home" or a slogan can lead an agent to pick the wrong page or give up. - Fix the templates for product, category and policy pages first, then hand-write titles and descriptions for your home page and top categories. Before an AI agent reads a page in full, it reads the labels: the title, the description and the headings. They're how it decides whether a page answers the question it was asked, and how it finds the right part of a long page. They're also the parts of a page most often left on autopilot. ### Why the labels matter more to agents than to people A person landing on your page takes in the design at a glance: the hero image, the logo, the layout. An agent sees none of that. It sees text, in order, and leans heavily on the few pieces of text that say what the page is. - **The `<title>`** is the page's name. It shows in browser tabs and search results, and it's often the first thing an agent records about a page it has visited. - **The meta description** is a one or two sentence summary. Search engines and AI search tools may show it, quote it, or use it to decide whether a page is worth opening. - **Headings** (`<h1>` to `<h6>`) are the outline. An agent looking for "delivery times" on a long FAQ uses them to jump to the right section. - **The `lang` attribute** on the `<html>` tag says what language the page is in, so the agent reads and quotes it correctly. If you've done SEO work, most of this will be familiar. The difference is that an agent isn't ranking your page among ten others; it's trying to complete a task, and a vague label can mean it picks the wrong page or gives up. There's more on that shift in [SEO got you found. Agent readiness gets you chosen](https://ghostagentlab.com/blog/seo-to-agent-readiness/). ### What AgentScore checks The "Clear page title, description, and headings" check, in the Readability category, reads your home page's HTML before any JavaScript runs and looks for four things: | Looks for | Counts as missing when | | --- | --- | | A `<title>` | There's no title tag, or it's empty | | A meta description | There's no `<meta name="description">`, or it's empty | | An `<h1>` heading | There's no h1 in the HTML (one added later by JavaScript doesn't count) | | A `lang` attribute | The `<html>` tag has no lang | All four present is a pass, and the result shows your title. One or two missing is a warning. It fails if there's no title at all, or if three or more are missing. The check confirms these things exist; it can't tell whether the wording is helpful. The rest of this guide is about that part. ### Writing titles agents can use A good title says what the page is and whose it is, specific enough to tell it apart from every other page on your site. | Page | Weak | Better | | --- | --- | --- | | Home page | Home | Northwind Coffee: specialty coffee beans and brewing gear | | Product | Shop | Ethiopia Yirgacheffe whole bean coffee, 340 g \| Northwind Coffee | | Category | Collection | Pour-over coffee kits \| Northwind Coffee | | Policy | Info | Shipping rates and delivery times \| Northwind Coffee | - **Put the specific part first.** The product or topic, then your brand. - **Make every title unique.** If twenty pages share a title, an agent comparing them has nothing to go on. - **Include the variant or size when it matters.** "340 g" or "Pro plan" is exactly what an agent needs to match a request. - **Skip the slogans.** "Where every cup tells a story" says nothing about what's on the page. ### Writing descriptions that summarize the page Write the meta description as the answer to "what will I find here?" One or two plain sentences: what the page offers, for whom, and anything that sets it apart, such as price range, delivery area or a free trial. ``` <meta name="description" content="Light-roast single-origin coffee from Yirgacheffe, Ethiopia, with notes of jasmine and lemon. Roasted to order, 340 g bag, ships free in the US over $40."> ``` Avoid repeating the title, stacking keywords, or reusing one description across many pages. If a platform fills it in automatically from the first lines of the page, check that those lines are useful. Cookie notices and "Free shipping this week" banners make poor summaries. ### Using headings as an outline Headings should read like a table of contents. Someone reading only your headings should understand what the page covers and where each part is. - **One h1 per page** that names the page's subject, usually close to the title. The check only needs at least one, but more than one makes the page's subject less clear. - **h2s for main sections, h3s inside them.** Don't skip levels or pick a heading level for its size; use CSS for size. - **Make headings descriptive.** "Delivery times by region" beats "Good to know". On FAQ pages, the question itself makes a good heading. - **Use real heading tags.** Bold text styled to look like a heading isn't one. Agents, and screen readers, can't see the styling. - **Keep headings in the HTML.** If your page builder adds them with JavaScript, many agents won't see them. See [our guide to JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). A product page outline might look like this: ``` h1 Ethiopia Yirgacheffe whole bean coffee h2 Tasting notes h2 Brewing suggestions h2 Shipping and returns h2 Reviews ``` ### Don't forget the lang attribute It's one attribute, and it's often missing from custom themes: `<html lang="en">`. Use the right code for each language version, such as `lang="fr"` or `lang="en-GB"`. It helps agents interpret the page, translate it correctly and quote it in the right language. Screen readers use it to choose a voice. ### Fixing it across a whole site 1. **Start with templates.** Titles, descriptions and headings on product, category and policy pages usually come from a theme template. Fix the pattern once and every page benefits. 2. **Feed templates real data.** Product name, variant, category and brand make good titles. Short product descriptions make good meta descriptions. 3. **Hand-write the important ones.** Your home page, top categories and policy pages deserve titles and descriptions written by a person. 4. **Crawl for duplicates.** Most SEO crawlers report missing and duplicate titles, descriptions and h1s across the whole site. 5. **Re-run [AgentScore](https://ghostagentlab.com/agentscore/)** to confirm the home page passes. > **Who owns this:** usually the SEO or content team, with a developer for template changes and the lang attribute. It's rarely a big project, which is why it's worth doing first. Clear labels are one part of a readable site. For the rest, see the [agent readiness guide](https://ghostagentlab.com/articles/agent-readiness/). ### Sources and further reading - [Influencing title links in Google Search](https://developers.google.com/search/docs/appearance/title-link) (Google Search Central) - [How to write meta descriptions](https://developers.google.com/search/docs/appearance/snippet) (Google Search Central) - [Headings (page structure tutorial)](https://www.w3.org/WAI/tutorials/page-structure/headings/) (W3C Web Accessibility Initiative) - [HTML Standard: The lang and xml:lang attributes](https://html.spec.whatwg.org/multipage/dom.html#the-lang-and-xml:lang-attributes) (WHATWG) --- ## Prices AI agents can read > How to show prices AI agents read correctly: in the server HTML, with a clear currency, labelled sale prices, variant prices and one source of truth. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/machine-readable-prices/ ### Key takeaways - Agents read prices from the page text and from structured data, and both must be in the HTML your server sends. - Struck-through sale prices, unclear currencies and mismatched prices across page, feed and cart can lead agents to quote the wrong amount. - View the source of a product page, search for the price, and ask your developer to render it on the server if it's missing. "How much is it?" is one of the first things a shopper asks an AI assistant about a product. If an agent can't find your price, or finds two prices and can't tell which is real, it either guesses or moves on to a store where the answer is clear. Here's how to make your prices unmistakable. ### How agents read a price Agents find prices in two places. The first is the text of the page: the "$18.00" a shopper sees next to the add-to-cart button. The second is structured data: a block of schema.org JSON-LD in the HTML that states the price, currency and stock as plain values. Good stores have both, and they agree. Both have to be in the HTML your server sends. Many agents, and most AI search crawlers, read that HTML without running JavaScript. If your theme loads prices from an API after the page appears, those agents see a product with no price. To a shopper asking an assistant to compare three coffee grinders, that's a product that doesn't make the list. ### What AgentScore checks Two Readability checks cover prices. | Check | What it looks at | | --- | --- | | Prices are in the page HTML | Your pricing page, or a product page if there's no pricing page. It reads the text of the HTML before JavaScript runs and looks for something that's clearly a price: an amount with a currency symbol ($, €, £, ¥, ₹) or a currency code such as USD, EUR, GBP, CAD or AUD. It passes if it finds one. It warns if prices only appear after JavaScript runs in a real browser, or if it can't find a price at all. | | Product pages give price and stock in a form agents can read | A product page's HTML, for schema.org `Product` or `ProductGroup` data with an offer that gives a price, a currency and availability. It passes with all three, warns if any are missing, if there's only microdata, or if the data is only added by JavaScript, and fails if there's none. Stores only. | AgentScore finds these pages from links on your home page, then from your [sitemap](https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/), then a common address such as `/pricing`. If it can't find one, the checks are marked "not tested" rather than failed. This guide focuses on the visible price and keeping everything consistent. For a full walk-through of product structured data, see [Product data that AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). ### Put the price in the HTML Open a product page, right-click and choose View Page Source (not Inspect, which shows the page after JavaScript). Search for the price. If it's there, agents can read it. If it isn't, the fix belongs to your developer or platform: render the price on the server, as part of the page, and let JavaScript update it afterwards if a shopper changes options. ``` <p class="price"> <span>$18.00</span> <span>USD</span> </p> ``` For software and services, the same applies to your pricing page: each plan's price, billing period and what it includes should be readable text, not an image or a widget that loads later. If you don't publish prices, say so in plain words, for example "Pricing is quoted per project. Contact sales for a quote." Then an agent can tell people how to get one, instead of reporting that it couldn't find any. ### Make the currency unambiguous "$18" means different things in the US, Canada and Australia. A person infers the currency from where they are. An agent may be working for someone elsewhere, fetching your page from a data center in another country. - **Use a currency code or prefix when it could be unclear.** "US$18.00", "$18.00 USD" or "CA$24.00" leaves no doubt. - **Always set `priceCurrency` in structured data** using the three-letter ISO 4217 code, such as `USD` or `EUR`. - **Be careful with automatic currency switching.** If you change the currency based on the visitor's location, an agent may see a different currency from its user. Give each currency a stable URL or a visible switcher, and keep the currency in the page text and the data consistent. - **Say whether tax is included.** "Includes VAT" or "plus sales tax" changes the answer to "how much is it?". ### Sale prices and "from" prices Discounts are where agents most often misread prices. A page showing "$24.00 $18.00" with the first one struck through looks obvious to a person, because they can see the line through it. An agent reading text may see two prices side by side. - **Label both prices in words.** "Was $24.00, now $18.00" reads correctly in any form. - **Mark the old price up as removed.** Use `<del>` or `<s>` for the original price, not just a CSS line-through. - **Put the price the customer will pay in structured data.** The offer's `price` should be the current sale price. If the sale ends on a known date, `priceValidUntil` says so. - **Make "from" prices explicit.** "From $18.00" on a product with several sizes should say what the $18.00 buys, such as "From $18.00 (340 g)". ``` <p class="price"> Was <del>$24.00</del>, now <strong>$18.00 USD</strong> <span>Sale ends October 31</span> </p> ``` ### Products with variants When sizes, colors or pack sizes have different prices, agents need to know which price goes with which option. A single price that changes with JavaScript when a shopper picks a size is invisible to an agent that doesn't click. - **List variant prices in the HTML**, for example in the option labels: "340 g, $18.00" and "1 kg, $44.00". - **Describe each variant in structured data.** Schema.org's `ProductGroup` with `hasVariant` gives each option its own offer with its own price and availability. AgentScore reads offers inside variants too. - **Give variants their own URLs** where your platform supports it, so an agent can link a shopper to exactly the option they asked about. ### Keep every price consistent An agent may see your price in several places: the page text, the JSON-LD, your product feed in Google Merchant Center or other shopping channels, and the cart. If they disagree, the agent can't know which is right. It may quote the wrong one, or treat your store as unreliable. Search engines may ignore structured data that doesn't match the page. > **One source of truth:** generate the visible price, the structured data and your product feeds from the same data, so a price change updates all of them together. Hand-entered prices in a separate feed or a hard-coded JSON-LD block are the usual cause of mismatches. Unit prices help too. For groceries, coffee, cosmetics and anything sold by weight or volume, a unit price such as "$5.29 per 100 g" lets an agent compare fairly across pack sizes. ### Your checklist 1. View the source of a product page and a pricing page. Is the price there? 2. Is the currency clear without knowing where the visitor is? 3. Do sale prices say which price is current, in words? 4. Does each variant's price appear in the HTML or the structured data? 5. Do the page, the JSON-LD, the feed and the cart all show the same price? 6. Run [AgentScore](https://ghostagentlab.com/agentscore/) and check both price checks pass. Prices are one of the facts agents most need to get right before they recommend you or buy from you. For the step after that, getting the item into a basket and through checkout, see [our guide to agent-ready checkout](https://ghostagentlab.com/articles/agent-ready-checkout/). ### Sources and further reading - [Offer (schema.org type)](https://schema.org/Offer) (schema.org) - [Product variant structured data (ProductGroup, Product)](https://developers.google.com/search/docs/appearance/structured-data/product-variants) (Google Search Central) - [General structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies) (Google Search Central) - [HTML Standard: The del element](https://html.spec.whatwg.org/multipage/edits.html#the-del-element) (WHATWG) --- ## Shipping, returns and FAQ pages AI assistants can quote > How to write shipping, returns and FAQ pages that AI assistants can find, read and quote accurately to shoppers before they buy. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/policy-pages-ai-assistants/ ### Key takeaways - AI assistants may now read your shipping, returns and FAQ pages for shoppers, so whatever they say becomes your answer. - Vague wording, policies in images or PDFs, and pages that contradict each other lead assistants to misquote you or decline to answer. - Ask a few AI assistants your most common support questions about your store, then rewrite the pages behind wrong or hesitant answers. "Do they ship to Canada?" "Can I return it if it doesn't fit?" "How long does delivery take?" These are the questions people ask AI assistants just before they buy. The answer comes from your shipping, returns and FAQ pages, quoted word for word or summarized. If those pages are vague, hidden or contradictory, the assistant gets it wrong, or tells the shopper it isn't sure. ### Why policy pages matter more now Policy pages used to be read by a small share of careful shoppers, and by your support team. Now an assistant may read them on behalf of every shopper who asks a question about delivery or returns. Whatever it says becomes your answer, with your name on it. That has two consequences. A clear policy page means accurate answers and confident shoppers. A muddled one means an assistant may misquote your returns window, invent a shipping rate, or decline to answer and suggest a store whose terms it could read. Neither shows up in your analytics as an error. It shows up as sales that didn't happen, or customers who expected something you never promised. AgentScore doesn't have a dedicated check for policy pages yet, so this guide is about what you can do and test yourself. ### Write so each sentence can be quoted on its own Assistants often lift a sentence or two out of a page. Write so that any sentence still makes sense when it's quoted without its neighbors. | Hard to quote | Easy to quote | | --- | --- | | We're happy to help with returns within the usual period, subject to the conditions above. | You can return unopened items within 30 days of delivery for a full refund. | | Shipping is fast and affordable. | US orders ship in 1 to 2 business days. Standard delivery costs $5.95 and takes 3 to 5 business days. Orders over $40 ship free. | | International shipping may be available. | We ship to the US and Canada. We don't ship to other countries yet. | | See exceptions. | Opened coffee and gift cards can't be returned. | - **Use numbers, not adjectives.** "30 days", "$5.95", "3 to 5 business days". "Fast" and "generous" can't be quoted usefully. - **Name the thing in every sentence.** "Returns are free" works alone. "They're free" doesn't. - **State the conditions next to the rule.** If the 30 days starts from delivery, not from the order date, say so in the same sentence. - **Say what you don't do.** Countries you don't ship to, items that can't be returned, services you don't offer. Assistants are often asked exactly these questions, and silence leaves them guessing. ### Structure the page around real questions Organize each page by the questions customers actually ask. Your support inbox and chat logs are the best source. Use those questions, or close versions of them, as headings, and answer directly underneath. That gives an agent a clear outline to search, and the heading itself matches the question it was asked. Our guide to [titles, descriptions and headings](https://ghostagentlab.com/articles/titles-descriptions-headings-ai/) covers how to build that outline. A shipping page might look like this: ``` h1 Shipping and delivery h2 Where do you ship? h2 How much does shipping cost? h2 How long does delivery take? h2 Can I track my order? h2 What if my order arrives damaged? ``` Use plain HTML tables for anything with more than two variables, such as rates by region and speed. A table reads clearly as text; a rate chart saved as an image doesn't read at all. ### Make sure agents can reach and read the page - **Put the text in the HTML.** Accordions are fine if the answers are already in the page and just hidden until clicked. They're a problem if each answer is loaded by JavaScript when opened. See [our guide to JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). AgentScore's JavaScript check looks at your home page, so check policy pages yourself with View Page Source. - **Avoid PDFs and images for policies.** Some agents can read PDFs, many won't bother, and text in images is invisible to most. - **Don't hide policies inside a chat widget.** A help bot can be useful for people, but agents need a page with a URL they can read. - **Link to them from every page.** Footer links to Shipping, Returns, FAQ and Contact, in plain HTML, are what agents follow. - **List them in your sitemap and llms.txt.** Your [sitemap](https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/) helps crawlers find them, and an [llms.txt file](https://ghostagentlab.com/articles/llms-txt/) can point agents straight at them with a one-line description each. - **Don't block them.** Check that robots.txt and bot protection let AI assistants read help pages. See [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). ### Keep one version of the truth Contradictions are the fastest way to an inaccurate answer. If the FAQ says 30-day returns, the returns page says 14 days and a product page says "free returns for 60 days", an assistant has to pick one, and it may not pick the one you meant. 1. **Choose a home for each policy.** The returns page owns the returns policy. The FAQ summarizes it and links there. 2. **Search your site for old terms** after any policy change: old return windows, old free-shipping thresholds, discontinued services. 3. **Match your structured data.** If your product markup includes return or shipping details, update it with the page. See [Product data that AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). 4. **Show a "last updated" date** at the top of each policy page, so agents and people can tell how current it is. 5. **Handle temporary changes explicitly.** For holiday cut-off dates or a carrier delay, add a dated note at the top ("Orders placed after December 18 may arrive after December 25") and remove it when it no longer applies. ### Different terms for different places or products If your policies vary by country, membership or product type, say so clearly and early. One page per region, or a clearly labelled section for each, works better than footnotes. Name the region in the heading: "Returns from Canada", not "Other regions". If marketplace sellers on your site set their own policies, say that too, and say where the shopper can find them. > **Test it the way your customers will:** ask two or three AI assistants the questions your support team hears most, naming your store. "What's Northwind Coffee's return policy?" "Does Northwind ship to Canada, and how much does it cost?" Compare the answers with your actual policy. Where they're wrong or hesitant, the page that should have answered is the one to rewrite. ### A short checklist - Every policy has numbers, conditions and exceptions in plain sentences. - Headings are the questions customers ask. - The text is in the HTML, not in images, PDFs or JavaScript-loaded panels. - Shipping, Returns, FAQ and Contact are linked from every page and listed in the sitemap. - No two pages disagree, and each policy page shows when it was last updated. Policy pages are where an assistant's answer turns into a sale or a lost one. Once they're clear, run [AgentScore](https://ghostagentlab.com/agentscore/) to check the rest of your site, and read the [agent readiness guide](https://ghostagentlab.com/articles/agent-readiness/) for the bigger picture. --- ## Business and store details AI agents can trust: Organization and LocalBusiness data > How to describe your business, contact points, opening hours and return policy with Organization and LocalBusiness JSON-LD that AI agents can trust. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/organization-structured-data/ ### Key takeaways - Describing your business in structured data gives agents one trusted summary of who you are, how to reach you and your return policy. - If the phone number, address, hours or return window in the markup differ from your pages, agents can't tell which is right. - Ask customer service which contact details and hours to share, then have the markup generated from the same settings as your footer. Before an AI agent recommends a store, it wants to know who it's dealing with: the business name, how to contact it, where it ships from, when it's open, and what happens if something goes back. Organization data puts those facts in one place, in a form an agent can read without piecing them together from your footer. ### Why business details matter to AI agents People ask AI assistants questions like "is this a real shop?", "do they have a phone number?", "are they open on Sunday?" and "can I return it if it doesn't fit?". To answer, an agent has to find those facts somewhere on your site. Usually they're scattered: the phone number in the footer, the address on a contact page, the returns window in a policy page, the opening hours in an image. An agent can often work it out, but every guess is a chance to get it wrong or to give up and recommend someone whose details are easier to confirm. Structured data about your business gives it one trusted summary, on the page it's most likely to visit first: your home page. Product markup describes what you sell. Organization markup describes who is selling it. Agents use both. For products, see [product data that AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). ### Pick the right type Schema.org has a family of types for businesses. Use the most specific one that's true. | Type | Use it when | | --- | --- | | `Organization` | The general case: any company, brand or nonprofit | | `OnlineStore` | You sell online and have no shop customers can walk into. It's a more specific kind of `Organization` | | `LocalBusiness` or a subtype such as `Store` | Customers can visit you at a physical address, such as a shop, showroom or clinic | If you have an online store and physical shops, describe the company as an `Organization` or `OnlineStore` on the home page, and give each shop its own `LocalBusiness` (or `Store`) markup on its own location page, with its own address and hours. ### A complete example Here's JSON-LD for the home page of a fictional online coffee store. It goes in the page's HTML, inside the `<head>` or `<body>`: ``` <script type="application/ld+json"> { "@context": "https://schema.org", "@type": "OnlineStore", "@id": "https://northwind.example/#organization", "name": "Northwind Coffee", "legalName": "Northwind Coffee Ltd", "url": "https://northwind.example/", "logo": "https://northwind.example/images/logo.png", "description": "Specialty coffee roasted to order and shipped across the US.", "email": "help@northwind.example", "telephone": "+1-555-010-0199", "address": { "@type": "PostalAddress", "streetAddress": "120 Harbor Street", "addressLocality": "Portland", "addressRegion": "OR", "postalCode": "97201", "addressCountry": "US" }, "contactPoint": [{ "@type": "ContactPoint", "contactType": "customer service", "email": "help@northwind.example", "telephone": "+1-555-010-0199", "availableLanguage": ["en"], "hoursAvailable": { "@type": "OpeningHoursSpecification", "dayOfWeek": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"], "opens": "09:00", "closes": "17:00" } }], "sameAs": [ "https://www.instagram.com/northwindcoffee.example", "https://www.linkedin.com/company/northwind-coffee-example" ], "hasMerchantReturnPolicy": { "@type": "MerchantReturnPolicy", "applicableCountry": "US", "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow", "merchantReturnDays": 30, "returnMethod": "https://schema.org/ReturnByMail", "returnFees": "https://schema.org/FreeReturn", "merchantReturnLink": "https://northwind.example/returns" } } </script> ``` ### What each part tells an agent - **Name, legal name, URL and logo.** Confirms which business this is, and that this website belongs to it. Use the name customers know you by in `name`, and your registered company name in `legalName`. - **Contact points.** A `ContactPoint` for each way to reach you, with `contactType` saying what it's for ("customer service", "sales", "technical support"). Add the hours it's staffed and the languages it covers, so an agent can tell someone whether they'll get an answer today. - **Address.** A full `PostalAddress` answers "where are they based?" and helps an agent judge shipping times and which country's consumer rules apply. Only include it if you're happy for it to be quoted. - **`sameAs`.** Links to your official profiles elsewhere: social accounts, marketplace storefronts, a Wikipedia or business-register entry. They help an agent connect your site to what others say about you, and make it harder to confuse you with a similarly named business. - **Return policy.** `hasMerchantReturnPolicy` on the organization sets a default for everything you sell. Google added support for return policies at the organization level, so you don't have to repeat it on every product. Product-level policies can still override it where they differ. ### Opening hours for places customers visit For a shop, showroom or other location, the most common question is "are they open now?". Use `openingHoursSpecification` on the `LocalBusiness`, one entry per set of days with the same hours: ``` "openingHoursSpecification": [ { "@type": "OpeningHoursSpecification", "dayOfWeek": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"], "opens": "08:00", "closes": "18:00" }, { "@type": "OpeningHoursSpecification", "dayOfWeek": "Saturday", "opens": "09:00", "closes": "16:00" } ] ``` Leave out days you're closed. For holidays, add an entry with `validFrom` and `validThrough` dates, so an agent doesn't send someone to a locked door on December 25. Add `geo` coordinates and a `telephone` for each location too. ### Rules that keep it trustworthy - **Match what people see.** The phone number, address, hours and return window in your markup must match your contact and policy pages. If they disagree, an agent can't tell which is right, and may say so. - **One source of truth.** Generate the markup from the same settings as your footer and contact page, so a change in one place updates both. Hand-written markup goes stale the first time your hours change. - **Put it in the HTML.** Markup added by a tag manager or other JavaScript after the page loads is invisible to agents that don't run JavaScript. - **Describe it once.** Use a stable `@id` (like `https://northwind.example/#organization`) so other markup on your site, such as a product's `brand` or a `WebSite`'s `publisher`, can point to the same organization instead of repeating it. - **Keep the policy pages too.** Markup summarizes; your returns and shipping pages explain. Agents read both. See [policy pages AI assistants can quote](https://ghostagentlab.com/articles/policy-pages-ai-assistants/). ### What AgentScore checks The [AgentScore](https://ghostagentlab.com/agentscore/) check "Structured data describes your business and products" reads your home page's HTML for schema.org JSON-LD: - It **passes** when JSON-LD is in the HTML before any JavaScript runs, and lists the types it found, such as `Organization` and `WebSite`. - It **warns** if the JSON-LD only appears after JavaScript runs, or if the page only has microdata. - It **fails** if there's no schema.org structured data on the home page at all. The check confirms the markup is there and readable. It doesn't judge whether your hours or phone number are right, so validate the content yourself. ### How to add it 1. **See what you have.** View your home page's source and search for `application/ld+json`. Many platforms and themes already output a basic `Organization` block with just a name and logo. 2. **Collect the facts.** Ask customer service for the contact details and hours they actually want shared, and check the return window against your current policy. 3. **Extend the markup.** Your developer can extend the theme template, or an SEO app or plugin can add the fields. Prefer one that reads from your store settings. 4. **Validate.** Run the page through the [Schema Markup Validator](https://validator.schema.org/), and Google's Rich Results Test for the fields Google uses. 5. **Re-check after changes.** Add "update the structured data" to the checklist for changing hours, phone numbers or policies. Business details are one part of being readable to AI agents. For the rest, see the [agent readiness guide](https://ghostagentlab.com/articles/agent-readiness/), and for the details shoppers ask about most, [reviews and ratings AI agents can read](https://ghostagentlab.com/articles/reviews-ratings-ai-agents/). ### Sources and further reading - [Organization schema markup](https://developers.google.com/search/docs/appearance/structured-data/organization) (Google Search Central) - [OnlineStore (schema.org type)](https://schema.org/OnlineStore) (schema.org) - [Local business (LocalBusiness) structured data](https://developers.google.com/search/docs/appearance/structured-data/local-business) (Google Search Central) --- ## Product descriptions AI agents can compare > How to write product descriptions AI agents can compare: specs as text and tables, consistent attributes, units, materials and compatibility. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/product-descriptions-ai-agents/ ### Key takeaways - AI agents recommend the products they can confirm, so pages with clear facts like weight and size make the shortlist over vague copy. - Specs that live only in images, PDFs or script-loaded tabs are often invisible to agents, and claims like "ultralight" can't be compared. - Start with your best-selling category: agree the five to ten attributes shoppers compare, then fill them in text, with units, on every product. When someone asks an AI agent for "a waterproof hiking jacket under 500 grams that packs into its own pocket", the agent has to read your product pages and compare them with everyone else's. It can only recommend what it can confirm. Product descriptions written as facts, not mood, are what get you onto the shortlist. ### How AI agents compare products A person browsing can look at a photo, skim the copy and get a feel for a product. An AI agent works differently. It turns the shopper's request into requirements (waterproof, under 500 grams, packable, under $200) and then checks each product page for evidence that the product meets them. If your page says "featherlight protection for every adventure", the agent can't match that to "under 500 grams". If a competitor's page says "Weight: 410 g (size M)", it can. The agent recommends the product it can back up, and yours drops out, even if it's the better jacket. This doesn't mean brand voice has to go. It means the facts need to be on the page too, written so they can be read and compared on their own. ### Write specs as text, not images Agents read text. Specifications that live only in an image, a size-chart graphic or a downloadable PDF are often invisible to them. The same goes for details that only appear when someone clicks a tab or accordion, if that content is loaded by JavaScript rather than present in the HTML. (Content hidden by CSS until clicked is fine; it's still in the page.) Put the key facts in the page's HTML as plain text. A simple table works well, because each row is a clear pair of attribute and value: ``` <table> <caption>Specifications</caption> <tr><th scope="row">Weight</th><td>410 g (size M)</td></tr> <tr><th scope="row">Waterproof rating</th><td>20,000 mm hydrostatic head</td></tr> <tr><th scope="row">Shell fabric</th><td>100% recycled nylon, 3-layer</td></tr> <tr><th scope="row">Packed size</th><td>18 x 12 x 7 cm, packs into chest pocket</td></tr> <tr><th scope="row">Fit</th><td>Regular, room for a fleece underneath</td></tr> </table> ``` A definition list (`<dl>`) works just as well. What matters is that each fact has a label next to it, in text. ### Make attributes comparable Comparison only works when the same attribute is described the same way across your catalog. Agree on a set of attributes for each category and fill them for every product. | Do | Avoid | | --- | --- | | Weight: 410 g | Ultralight | | Battery life: up to 14 hours of video playback | All-day battery | | Capacity: 1.7 liters (7 cups) | Family size | | Dimensions: 60 x 40 x 25 cm (W x D x H) | Compact design | | Material: 100% organic cotton, 180 gsm | Premium soft fabric | | Fits: iPhone 15 and iPhone 16 (not Pro Max) | Fits most phones | Use the same attribute names, the same order and the same units on every product in a category. An agent comparing your three kettles should find "Capacity" in the same place on each page. ### Units, materials and measurements - **Always give the unit.** "Width: 60" means nothing. "Width: 60 cm" does. Where your customers use both systems, give both: "60 cm (23.6 in)". - **Say what was measured.** "Weight: 410 g (size M)" or "Battery life: up to 14 hours (video playback, 50% brightness)". Conditions turn a claim into a fact. - **Name the materials.** Give the fiber, metal or wood, the percentage in blends, and any certification by its proper name. "Vegan leather" is a category; "polyurethane-coated cotton" is a material. - **Give exact sizes.** Link to a size guide in HTML text, not only an image, with body measurements per size. - **State what's in the box.** Agents get asked "does it come with a charger?". Answer it in the description. ### Compatibility and fit For parts, accessories, consumables and refills, compatibility is the whole decision. A shopper asks "will this filter fit my Model X200?" and the agent needs a clear yes or no. - List every compatible model by its full name and model number, not "fits most models". - List known incompatible models where confusion is likely, such as the Pro version of a device. - Put the list on the product page in text, not only in a separate compatibility tool that needs a form to be filled in. In structured data, schema.org has properties such as `isAccessoryOrSparePartFor` and `isConsumableFor` that point from your product to the product it's used with. ### Add the facts to structured data Product JSON-LD is where agents look first for price and stock. It can carry descriptive attributes too, so they don't have to be read out of your layout: ``` { "@context": "https://schema.org", "@type": "Product", "name": "Ridgeline Packable Rain Jacket, Men's", "sku": "NW-RJ-M-BLU", "material": "100% recycled nylon", "color": "Blue", "size": "M", "weight": { "@type": "QuantitativeValue", "value": 410, "unitCode": "GRM" }, "additionalProperty": [ { "@type": "PropertyValue", "name": "Waterproof rating", "value": 20000, "unitText": "mm" }, { "@type": "PropertyValue", "name": "Packable", "value": true } ], "offers": { "@type": "Offer", "price": "189.00", "priceCurrency": "USD", "availability": "https://schema.org/InStock" } } ``` Use `additionalProperty` for attributes schema.org has no dedicated property for. The values must match what the page says. For the full product markup, including variants, returns and shipping, see [product data that AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). ### Avoid vague copy Marketing copy still has a job: it persuades the person who reads it. The problem is when it replaces facts instead of sitting beside them. Watch for these: - **Superlatives without evidence.** "Best-in-class", "industry-leading", "unbeatable". An agent can't verify them, so it ignores them. - **Relative claims.** "30% lighter" than what? Give the absolute number. - **Copy shared across a range.** If ten products share the same paragraph, an agent can't tell them apart. Lead with what makes each one different. - **Manufacturer text pasted unchanged.** It's often thin, and the same on every store selling that product. Add your own facts. A good pattern: one or two sentences of plain summary (what it is, who it's for, the one or two facts that matter most), then your brand copy, then the full specification table. ### What AgentScore checks [AgentScore](https://ghostagentlab.com/agentscore/) doesn't grade the wording of your descriptions. On a product page, it checks that schema.org Product data gives price, currency and availability ("Product pages give price and stock in a form agents can read"), and that prices are in the page HTML. To see how a real agent handles your descriptions, run a journey with [Ghost Agent](https://ghostagentlab.com/ghost-agent/), such as "find a waterproof jacket under 500 grams", and read the step-by-step replay. See [testing journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). ### Where to start 1. Pick your best-selling category and list the five to ten attributes shoppers compare. 2. Audit ten product pages: is each attribute there, in text, with a unit? 3. Fill the gaps in your product information system or platform, not in each page by hand, so the data reaches every channel. 4. Add the specification table to the product page template and the same attributes to your Product JSON-LD. 5. Repeat for the next category. Shoppers also lean on prices, variants and reviews when comparing; see [machine-readable prices](https://ghostagentlab.com/articles/machine-readable-prices/), [variant pickers AI agents can use](https://ghostagentlab.com/articles/variant-pickers-ai-agents/) and [reviews and ratings AI agents can read](https://ghostagentlab.com/articles/reviews-ratings-ai-agents/). ### Sources and further reading - [Product](https://schema.org/Product) (schema.org) - [PropertyValue](https://schema.org/PropertyValue) (schema.org) - [isAccessoryOrSparePartFor](https://schema.org/isAccessoryOrSparePartFor) (schema.org) --- ## Reviews and ratings AI agents can read > How to make reviews and ratings visible to AI agents: reviews in the HTML, AggregateRating and Review markup that follows the rules, and honest moderation. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/reviews-ratings-ai-agents/ ### Key takeaways - Agents use your star rating to rank options and your review text to answer shoppers' questions, but only if they can see them. - Many review apps load reviews with JavaScript, so agents that read the HTML see an empty space where your ratings should be. - Check a product page's source for your rating and reviews, and if they're missing, turn on your review provider's server-side output. "Which of these has the best reviews?" is one of the most natural questions to ask an AI shopping assistant. To answer it, the agent has to find your reviews, read them and trust them. Many stores load reviews in a way agents never see, or mark them up in ways that break the rules. Here's how to get it right. ### Why reviews matter to AI agents AI agents use reviews in two ways. The rating is a quick signal when they rank options: a 4.7 from several hundred reviews reads differently from a 4.7 from three. The review text answers specific questions the product description doesn't: "does it run small?", "is it loud?", "how long did the battery last after a year?". If an agent can't see your reviews, it has nothing to weigh. It may fall back on reviews it finds elsewhere, or simply prefer a competitor whose ratings it can confirm. ### Put reviews in the HTML Most stores use a review app or service. Many of these load reviews with JavaScript after the page opens, from a separate server. A person sees them a moment later. An AI agent that reads the HTML without running JavaScript sees an empty space where the reviews should be, and many agents work that way. - **Check the source.** Open a product page, view the page source (not the browser's inspector, which shows the page after JavaScript), and search for a phrase from one of your reviews. If it isn't there, agents that skip JavaScript don't see it. - **Ask your provider about server-side output.** Many review platforms offer a way to include the rating and the most recent or most helpful reviews in the page HTML, through a platform integration or an API. Turn it on if yours does. - **At minimum, render the summary.** The average rating and review count, in text near the product name, plus a handful of reviews. "Rated 4.6 out of 5 from 318 reviews" is easy for any agent to read. - **Link paginated reviews.** If reviews run to several pages, use ordinary links between them so agents can follow them. For the wider problem of content that only appears after JavaScript runs, see [why AI agents can't see JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). ### Mark up ratings and reviews Schema.org gives agents the same facts in a standard form. On a product page, add `aggregateRating` for the summary and `review` for individual reviews, inside your Product JSON-LD: ``` { "@context": "https://schema.org", "@type": "Product", "name": "Ethiopia Yirgacheffe Whole Bean Coffee, 340 g", "aggregateRating": { "@type": "AggregateRating", "ratingValue": "4.6", "bestRating": "5", "reviewCount": "318" }, "review": [{ "@type": "Review", "author": { "@type": "Person", "name": "Dana R." }, "datePublished": "2026-09-14", "reviewRating": { "@type": "Rating", "ratingValue": "5", "bestRating": "5" }, "reviewBody": "Bright and floral as a pour-over. Arrived two days after roasting." }], "offers": { "@type": "Offer", "price": "18.00", "priceCurrency": "USD", "availability": "https://schema.org/InStock" } } ``` Use `reviewCount` for reviews with text, or `ratingCount` if you count star-only ratings too. Say what scale you use with `bestRating` if it isn't 1 to 5. ### The rules for review markup Search engines publish rules for review markup, and following them is the safest guide for AI agents too. The key ones from [Google's review snippet guidelines](https://developers.google.com/search/docs/appearance/structured-data/review-snippet): - **Only mark up reviews people can see.** Every review and rating in the markup should be readable on the same page. Markup for reviews hidden somewhere else, or that don't exist, breaks the rules and can get your structured data ignored. - **Mark up reviews of a specific thing.** A product, a recipe, a course, a book. Not a category page summing up many products. - **Don't add review markup about your own business on your own site.** Google calls reviews that a business hosts about itself, on its own site, "self-serving", and doesn't show review stars for them on `LocalBusiness` and `Organization` pages. The safe reading is: put ratings on your products, and leave your company-level reputation to independent review sites. - **Keep the numbers in step.** The rating and count in the markup should match what the page shows, and update when new reviews come in. Google's rules apply to its search results, not to every AI agent. But they describe what makes review data trustworthy, and agents that cross-check your markup against the page will notice the same problems. ### Honesty is the strategy AI agents read across many sources. If your site shows a perfect 5.0 and independent review sites say 3.2, the gap is itself a signal, and an agent may say so to the shopper. The only durable approach is reviews that reflect what customers actually think. - **Publish the negative reviews.** Hiding everything below four stars makes the remaining rating less believable, to people and to agents. In the US, the FTC's rule on fake reviews and testimonials covers suppressing negative reviews as well as faking positive ones. Check the rules where you sell. - **Label incentives.** If a reviewer got the product free or was rewarded for reviewing, say so next to the review. - **Mark verified buyers.** A "verified purchase" label, applied honestly, helps agents weigh reviews. - **Reply in public.** A clear reply to a complaint ("we've changed the packaging since March") is information an agent can pass on. - **Don't pool unrelated products.** Showing reviews of an old model, or of the whole range, on a new product's page misleads. Pool reviews only across true variants, like sizes and colors of the same item. ### What AgentScore checks [AgentScore](https://ghostagentlab.com/agentscore/) doesn't have a dedicated reviews check. Its structured data checks look for schema.org JSON-LD on your home page, and for Product data with price, currency and availability on a product page, and flag markup that only appears after JavaScript runs. If your review app adds its markup with JavaScript, the same problem applies to your ratings. To see what an agent actually makes of your reviews, run a [Ghost Agent](https://ghostagentlab.com/ghost-agent/) journey such as "find the best-rated coffee under $20 and add it to the cart" and read the replay. ### A short checklist 1. View the source of a product page. Are the rating, review count and some review text in the HTML? 2. If not, turn on your review provider's server-side or SEO output, or ask your developer to render a summary. 3. Check the Product JSON-LD includes `aggregateRating` that matches the page. 4. Remove any rating markup about your business as a whole from your own site. 5. Review your moderation policy: are you publishing negative reviews and labeling incentives? 6. Validate with Google's Rich Results Test. Reviews work best alongside clear facts. See [product descriptions AI agents can compare](https://ghostagentlab.com/articles/product-descriptions-ai-agents/) and [product data that AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). ### Sources and further reading - [Review snippet (Review, AggregateRating) structured data](https://developers.google.com/search/docs/appearance/structured-data/review-snippet) (Google Search Central) - [AggregateRating](https://schema.org/AggregateRating) (schema.org) - [Review](https://schema.org/Review) (schema.org) --- ## Canonical URLs and duplicate pages for AI agents > How to stop AI agents reading the same page at many URLs: rel=canonical, parameter and variant URLs, http, https and www redirects, and clean sitemaps. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/canonical-urls-ai-agents/ ### Key takeaways - Stores show the same product at many web addresses, and naming one main address helps AI agents read and repeat the right version. - Without it, agents may treat one product as several, quote an old price, or send shoppers to a page with a stale campaign tag. - Set a canonical link on every page, redirect old http and www versions in one step, and list only main addresses in your sitemap. Most online stores show the same product at many different addresses: with tracking tags, with a color selected, under two categories, on http and https. People don't notice. AI agents do, because each address can look like a different page. Canonical URLs tell them which one is the real one. ### Why duplicate URLs confuse AI agents When an AI agent reads your site, or when an AI search service builds the index an assistant draws on, it meets your pages by URL. If the same jacket appears at five URLs, it may: - treat them as five products, and compare your jacket with itself; - quote a price or stock level from an old copy it read weeks ago; - link a shopper to a version with a stale campaign tag, an odd filter or a variant they didn't ask for; - spend its limited time on your site reading copies instead of finding new products. The fix is not to make duplicates disappear, which is often impossible, but to say clearly which URL is the main one, and to make every other signal agree. ### Where duplicates come from | Source | Example | | --- | --- | | Tracking parameters | `/products/rain-jacket?utm_source=newsletter` | | Sort and filter parameters | `/jackets?sort=price&color=blue` | | Variant parameters | `/products/rain-jacket?variant=4417` | | Category paths | `/mens/jackets/rain-jacket` and `/sale/rain-jacket` | | Protocol and host | `http://` and `https://`, with and without `www.` | | Trailing slashes and case | `/Rain-Jacket/` and `/rain-jacket` | ### Set a canonical URL on every page A canonical link is one line in the page's `<head>` that names the preferred URL for the content: ``` <link rel="canonical" href="https://northwind.example/products/rain-jacket"> ``` Put it on every indexable page, including on the canonical page itself (a self-referencing canonical). Rules that keep it reliable: - **Use the full, absolute URL.** Include `https://` and the host, not just the path. - **Put it in the HTML.** A canonical tag added by JavaScript may never be seen by agents and crawlers that read the raw page. - **One per page.** Two different canonical tags on a page is a common theme or app conflict. Search engines may ignore both. - **Point to a page that works.** The canonical should return 200, not redirect, 404 or carry a `noindex`. - **Don't canonicalize everything to the home page.** Each distinct page should point at itself, or at its true duplicate. Search engines treat `rel="canonical"` as a strong hint rather than a command, and the same is likely true of AI services that build on them. That's why the other signals below need to agree with it. ### Parameter URLs Tracking parameters like `utm_source` and click IDs never change the content, so the page should always name the clean URL as canonical. Sort orders and session IDs are the same. Filters are a judgment call. A filtered listing such as "blue jackets" may be worth keeping as its own page if people search for it. If so, give it a clean path (`/jackets/blue`), its own title and a self-referencing canonical. Most combinations of filters, though, should point back to the unfiltered category. For more on filters agents can use, see [site search and filters for AI agents](https://ghostagentlab.com/articles/site-search-filters-ai-agents/). ### Product variants Variants are where stores most often go wrong. There are two reasonable patterns: - **One page for the product.** Every variant URL (`?variant=4417`, `?color=blue`) has a canonical pointing to the main product URL. This suits products where variants differ only in size or color and share a description. Make sure the main page lists every variant's price and availability, ideally in `ProductGroup` markup, so the agent doesn't lose them. - **A page per variant.** Each variant has its own URL with a self-referencing canonical, its own title ("Rain Jacket, Men's, Blue") and its own price. This suits variants that are really different products, with different specs or prices. What doesn't work is a mix: variant URLs that canonicalize to the main page, while the main page only shows the default variant's price and stock. An agent is then told the blue jacket is the same page as the red one, but can't find the blue one's details. See [size, color and variant pickers AI agents can use](https://ghostagentlab.com/articles/variant-pickers-ai-agents/). ### http, https and www: redirect, don't just canonicalize For duplicates that are purely technical, a canonical tag isn't enough. Send a permanent (301 or 308) redirect so that every visitor, person or agent, lands on one version: - `http://` redirects to `https://`. - Pick `www.northwind.example` or `northwind.example`, and redirect the other. - Pick trailing slash or no trailing slash, and redirect the other. - Redirect old product URLs to their replacements after a migration, not to the home page. Redirect in a single hop. Chains (http to https, then to www, then to a trailing slash) slow every request down, and some clients stop following after a few hops. Avoid redirects that depend on JavaScript or a meta refresh; agents that don't run scripts get stuck on the first page. And make sure region or language redirects don't trap agents, as covered in [international sites for AI agents](https://ghostagentlab.com/articles/international-sites-ai-agents/). ### Make every signal agree Agents and crawlers look at more than the canonical tag. Each of these should use the same, canonical URL: - **Your XML sitemap.** List only canonical URLs that return 200. No parameter URLs, no redirecting URLs, no http versions. See [XML sitemaps for AI agents](https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/). - **Internal links.** Navigation, product grids and breadcrumbs should link to the canonical URL directly, not to a version that redirects or carries parameters. - **Structured data.** The `url` in your Product and Offer JSON-LD should match the canonical. - **`og:url` and feeds.** Social tags and product feeds should use the same address. - **hreflang tags**, if you have them, should point to canonical URLs of each language version. ### What AgentScore checks [AgentScore](https://ghostagentlab.com/agentscore/) doesn't have a dedicated canonical check. The "Valid sitemap" check confirms you have a sitemap it can read (found through robots.txt, `/sitemap.xml` or `/sitemap_index.xml`) and counts its entries; it doesn't judge whether each URL is canonical. AgentScore also uses your sitemap and home page links to find a product page to test, so a sitemap full of redirecting or parameter URLs makes that harder for any agent. ### A quick audit 1. Open a product page with `?utm_source=test` on the end. View the source and check the canonical points to the clean URL. 2. Do the same for a variant URL and a filtered category. Is the result the pattern you intended? 3. Type `http://` and the non-preferred host into a browser. Do they each redirect, in one hop, to the right page? 4. Open your sitemap and spot-check ten URLs: do they all return 200, and match their own canonical? 5. Check your main navigation links for parameters or redirects. Canonical URLs are housekeeping, but they decide which version of your page agents read and repeat. For the bigger picture, see the [agent readiness guide](https://ghostagentlab.com/articles/agent-readiness/). ### Sources and further reading - [How to specify a canonical with rel="canonical" and other methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) (Google Search Central) - [Redirects and Google Search](https://developers.google.com/search/docs/crawling-indexing/301-redirects) (Google Search Central) - [RFC 6596: The Canonical Link Relation](https://www.rfc-editor.org/rfc/rfc6596.html) (IETF) --- ## Languages, currencies and regions: international sites for AI agents > How to make international stores work for AI agents: lang and hreflang, unambiguous currencies, region redirects that don't trap agents, and shipping countries. Published October 9, 2026 by Ghost Agent Labs · Readability · https://ghostagentlab.com/articles/international-sites-ai-agents/ ### Key takeaways - If you sell abroad, AI agents need to know each page's language, the currency of every price and whether you deliver to the shopper. - Automatic redirects based on location can trap agents in the wrong country store, because they often run from data centers elsewhere. - Let every country address open directly without forced redirects, show currency codes where symbols are ambiguous, and list the countries you ship to. If you sell in more than one country, an AI agent needs to know which version of your site it's reading: what language it's in, which currency the prices are in, and whether you'll deliver to the shopper's address. Get this wrong and an agent may quote a euro price to someone in Ohio, or never get past your country picker at all. ### What can go wrong International sites are built for people, who can see a flag in the corner, read a price symbol and click "change country". AI agents work from the page's code and text, often from a server in another country, and usually without the browser settings that a person's visit carries. Common failures: - The agent is redirected to the wrong country store based on where its server is, and can't get back. - A price shows "$" with no sign of whether it's US, Canadian or Australian dollars. - Three language versions of the same page look like three unrelated pages, or like duplicates. - The agent tells a shopper a product is available, when you don't ship to their country. ### Declare the language with the lang attribute Every page should say what language it's in, on its `<html>` tag: ``` <html lang="en-GB"> ``` Use a language code, optionally with a region: `en`, `en-GB`, `fr-CA`, `de-CH`. It tells agents, screen readers and translation tools how to read the page. Make sure it changes with the language version. A French page that still says `lang="en"`, because the theme hard-codes it, is surprisingly common. The [AgentScore](https://ghostagentlab.com/agentscore/) check "Clear page title, description, and headings" flags a missing `lang` attribute on your home page; see [titles, descriptions and headings](https://ghostagentlab.com/articles/titles-descriptions-headings-ai/). ### Connect language and country versions with hreflang `hreflang` links tell agents and search engines that several pages are versions of the same content for different languages or regions, and which one is for whom. Each version lists all the others, and itself: ``` <link rel="alternate" hreflang="en-us" href="https://northwind.example/us/products/rain-jacket"> <link rel="alternate" hreflang="en-gb" href="https://northwind.example/uk/products/rain-jacket"> <link rel="alternate" hreflang="fr-fr" href="https://northwind.example/fr/produits/veste-pluie"> <link rel="alternate" hreflang="x-default" href="https://northwind.example/products/rain-jacket"> ``` - **Make it two-way.** If the US page lists the UK page, the UK page must list the US page. One-way links are usually ignored. - **Use canonical URLs.** Each `href` should be the canonical URL of that version, returning 200, not a redirect. See [canonical URLs and duplicate pages](https://ghostagentlab.com/articles/canonical-urls-ai-agents/). - **Don't canonicalize across languages.** The French page's canonical should point to the French page, not the English one. - **Add `x-default`.** It names the page for everyone else, typically a global version or a country picker that works without redirects. - **Use correct codes.** Language first, then optional region: `en-gb`, not `uk` or `gb-en`. You can put hreflang in the page `<head>`, in HTTP headers, or in your XML sitemap. Pick one method and keep it consistent. ### Region redirects that trap agents Many stores redirect visitors automatically based on their IP address or browser language. For people, it saves a click. For AI agents it can be a dead end. Agents often run from cloud data centers, so the location you detect is the server's, not the shopper's. An agent helping someone in Berlin may be sent to your US store, and if every attempt to open `/de/` bounces back to `/us/`, it can never read the German prices. - **Suggest, don't force.** Show a banner ("Looks like you're in Germany. Go to the German store?") instead of redirecting. - **Never redirect away from an explicit country URL.** If a request asks for `/de/products/rain-jacket`, serve it. - **Let the choice stick in the URL.** Country and language in the path or subdomain work for agents. A choice stored only in a cookie doesn't, because many agents don't keep cookies between requests. - **Don't block whole regions without a reason.** Blocking traffic from countries you don't sell to can also block the data centers AI agents run from. See [geo-blocking, VPN blocks and AI agents](https://ghostagentlab.com/articles/geo-blocking-ai-agents/). ### Make the currency unambiguous A symbol alone is often not enough. "$" is used by more than a dozen currencies, and "kr" by several. Show the currency code where there's any doubt (`US$189` or `189.00 USD`), and always put it in your structured data: ``` "offers": { "@type": "Offer", "price": "175.00", "priceCurrency": "GBP", "availability": "https://schema.org/InStock" } ``` `priceCurrency` takes a three-letter ISO 4217 code: `USD`, `GBP`, `EUR`, `CAD`. Each country version should show its own price and currency in both the page and the markup. Say whether prices include sales tax or VAT, since that differs by country. More in [machine-readable prices](https://ghostagentlab.com/articles/machine-readable-prices/). > **Watch out for converted prices.** If prices are converted on the fly with JavaScript, an agent that doesn't run JavaScript sees your base currency, while the shopper would have paid something else. Render each store's price in the HTML. ### Say where you ship "Do they deliver to Canada?" is a question an agent should be able to answer before sending someone to your checkout. Make it easy: - **A shipping page in plain text** listing the countries you ship to, with costs and delivery times for each. Not only a dropdown at checkout. - **Product-level restrictions on the product page.** If an item can't ship to some countries, say so on that page. - **Shipping details in structured data.** Schema.org's `shippingDetails` on an Offer can describe where you deliver, at what cost, and how long it takes: ``` "shippingDetails": { "@type": "OfferShippingDetails", "shippingDestination": { "@type": "DefinedRegion", "addressCountry": ["GB", "IE"] }, "shippingRate": { "@type": "MonetaryAmount", "value": "4.95", "currency": "GBP" }, "deliveryTime": { "@type": "ShippingDeliveryTime", "handlingTime": { "@type": "QuantitativeValue", "minValue": 0, "maxValue": 1, "unitCode": "DAY" }, "transitTime": { "@type": "QuantitativeValue", "minValue": 2, "maxValue": 4, "unitCode": "DAY" } } } ``` Returns can differ by country too; give each `MerchantReturnPolicy` its `applicableCountry`. See [Organization and LocalBusiness data](https://ghostagentlab.com/articles/organization-structured-data/) and [policy pages AI assistants can quote](https://ghostagentlab.com/articles/policy-pages-ai-assistants/). ### A checklist for each country version 1. The `<html lang>` attribute matches the page's language. 2. hreflang links connect every version both ways, with an `x-default`. 3. Each version's canonical points to itself. 4. Opening a country URL directly serves that country's page, with no forced redirect. 5. Prices are in the HTML, with an unambiguous currency, and `priceCurrency` matches. 6. The countries you ship to are listed in text, and in `shippingDetails` where you can. If your country stores run on separate domains, such as `northwind.example` and `northwind-uk.example`, scan each one with [AgentScore](https://ghostagentlab.com/agentscore/) to compare them. To check a real journey in another market, such as "find a rain jacket and get to checkout on the UK store", run it with [Ghost Agent](https://ghostagentlab.com/ghost-agent/) and read the replay. ### Sources and further reading - [Localized versions of your pages](https://developers.google.com/search/docs/specialty/international/localized-versions) (Google Search Central) - [Declaring language in HTML](https://www.w3.org/International/questions/qa-html-language-declarations) (W3C) - [priceCurrency](https://schema.org/priceCurrency) (schema.org) - [OfferShippingDetails](https://schema.org/OfferShippingDetails) (schema.org) --- ## Buttons, links and forms agents can use: a checklist > Browser agents navigate by the names and roles of controls. Fix icon-only buttons, fake buttons, unlabelled fields, blocking pop-ups and hover-only menus. Published October 8, 2026 by Ghost Agent Labs · Navigability · https://ghostagentlab.com/articles/agent-friendly-buttons-forms/ ### Key takeaways - Browser agents find their way by the names and roles of buttons, links and fields, so accessibility work also makes your site agent-ready. - Icon-only buttons, clickable boxes that aren't real buttons, unlabelled fields, covering pop-ups and hover-only menus are where agents most often get stuck. - Try reaching the menu, search, a product and the cart with the keyboard alone, because anywhere you get stuck, an agent probably does too. Browser agents don't see your page the way people do. They read its structure: what each button, link and form field is called, and what it does. A shopping-bag icon that a person recognizes instantly can be completely invisible to an agent. This checklist covers the fixes that matter most. ### How agents find their way around Most browser agents work from the page's *accessibility tree*, the same structured view of the page that screen readers use, often alongside a screenshot. In that tree, every element has a role (button, link, text box, heading) and a name ("Add to cart", "Search", "Email address"). The agent decides what to do by reading those names. That's good news: almost everything that makes a site work for agents also makes it work for people using screen readers and keyboards. If you've invested in accessibility, you're most of the way there. What an agent reads - `button` with no name (the search icon) - `button` with no name (the cart icon) - `heading` “Trail runner” - `text` “$129” - `generic`: the color swatches aren’t controls at all - `button` “Add to cart” *The same product page two ways. Icon-only buttons have no name, and swatches built from plain boxes aren’t controls, so the only thing an agent can use with confidence is the button with words on it.* ### 1. Give every button and link a name The most common failure is the icon-only control: a cart, search, menu or close button that shows an icon and has no text. ``` <!-- An agent sees: button, no name --> <button><svg>…</svg></button> <!-- An agent sees: button "Cart, 2 items" --> <button aria-label="Cart, 2 items"><svg aria-hidden="true">…</svg></button> ``` - Use visible text where you can. Use `aria-label` where the design calls for an icon alone. - For a linked image, the image's `alt` text becomes the link's name. - Make names specific. Ten links called "Learn more" or "Shop now" tell an agent nothing about where each goes; "Shop running shoes" does. ### 2. Use real buttons and links A `<div>` with a click handler looks like a button to a person but isn't one to an agent: it has no role, can't be focused, and often isn't recognized as clickable at all. ``` <!-- Looks clickable, isn't recognized as clickable --> <div class="btn" onclick="addToCart()">Add to cart</div> <!-- Recognized as a button by every agent --> <button type="button" onclick="addToCart()">Add to cart</button> ``` - Use `<a href>` for anything that goes to another page, and `<button>` for anything that does something on this page. - If you truly can't change the element, add `role="button"` and `tabindex="0"`, and handle the Enter and Space keys. - Make links real URLs. A link with `href="#"` and a script gives an agent nowhere to go if the script doesn't run. ### 3. Label every form field Agents fill in forms by matching what they've been asked to enter ("my email", "ship to 10001") to the field's label. Placeholder text is a weak substitute: it disappears once someone types, and not every agent reads it. ``` <label for="email">Email address</label> <input id="email" name="email" type="email" autocomplete="email" required> ``` - Connect each `<label>` to its field with `for` and `id`, or wrap the field in the label. - Use the right `type` (`email`, `tel`, `number`) and `autocomplete` values (`given-name`, `postal-code`, `address-line1`). They tell an agent exactly what each field wants. - Prefer native `<select>`, checkboxes and radio buttons over custom-built dropdowns. Custom size and color pickers are a frequent place agents get stuck. - Show errors as text next to the field and link them with `aria-describedby`, so an agent knows what to fix. A red border alone isn't enough. ### 4. Get pop-ups and banners out of the way Cookie banners, newsletter pop-ups and region selectors that cover the page on arrival are one of the biggest blockers. An agent has to work out how to dismiss them before it can do anything else, and if it can't, it's stuck. - Close and accept controls must be real buttons with clear names: "Accept all", "Reject all", "Close". - Keep consent banners small, at the edge of the screen, rather than covering the content. - Don't show sign-up pop-ups on arrival. If you must, wait until someone has engaged with the page. ### 5. Make menus work without hover Agents click; they don't hover. A menu that only opens when the mouse rests over it may never open for an agent, hiding every category inside it. - Make the top-level menu item a button that opens the menu on click, with `aria-expanded` showing whether it's open. - Make sure the top-level items are also real links to category pages, so there's always a route in. ### 6. Mark up the page's landmarks Landmarks tell an agent where the main content is, and where the navigation is, so it can skip straight to what matters. ``` <header>…</header> <nav aria-label="Main">…</nav> <main>…</main> <footer>…</footer> ``` Use one `<h1>` per page and headings in order, so the page has an outline an agent can follow. ### How to test it 1. **Tab through the page.** Use only the keyboard to reach the menu, search, a product, its options and the cart. Anywhere you get stuck, an agent probably does too. 2. **Look at the accessibility tree.** In Chrome's developer tools, the Accessibility pane shows each element's role and name. Look for buttons with no name. 3. **Run an accessibility checker**, such as Lighthouse in Chrome or axe, for a quick list of unnamed controls and unlabelled fields. 4. **Run [AgentScore](https://ghostagentlab.com/agentscore/).** It renders your home page in a browser and checks named buttons and links, real controls, form labels, landmarks and pop-ups that cover the page. To see where a real AI agent gets stuck on a journey, run a [Ghost Agent](https://ghostagentlab.com/ghost-agent/) test. > **Test the journeys, not just the page.** A home page can pass every check while the size picker on a product page or the checkout button stops agents cold. [Ghost Agents](https://ghostagentlab.com/ghost-agent/) run your key journeys, such as search, add to cart and sign-up, on a schedule, and alert you when one breaks. For the bigger picture, see [What is agent readiness?](https://ghostagentlab.com/articles/agent-readiness/) ### Sources and further reading - [WCAG 2.2](https://www.w3.org/TR/WCAG22/) (W3C) - [Button pattern (ARIA Authoring Practices Guide)](https://www.w3.org/WAI/ARIA/apg/patterns/button/) (W3C) - [HTML Standard: form control infrastructure (autofill)](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html) (WHATWG) --- ## Cookie banners and pop-ups that trap AI agents > How cookie banners, sign-up pop-ups and chat widgets block AI agents, what AgentScore checks, and how to design overlays agents can dismiss. Published October 9, 2026 by Ghost Agent Labs · Navigability · https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/ ### Key takeaways - A cookie banner or pop-up an agent can't dismiss ends its visit, and you never hear about it because the agent simply moves on. - The usual culprits are unlabelled close icons, fake buttons, reject options hidden in settings, and chat widgets sitting on top of "Add to cart". - Keep consent banners at the edge of the screen with named "Accept all" and "Reject all" buttons, and agree any change with your privacy owner. The first thing many AI agents see on a website isn't the home page. It's a cookie banner, a discount pop-up or a country selector sitting on top of it. A person closes it without thinking. An agent has to find the right button, work out what it means, and press it, and if it can't, the visit ends there. This guide covers the overlays that cause trouble, what AgentScore checks, and how to design them so they stay out of the way. ### Why overlays stop AI agents Browser agents, the kind that open a real browser and click through a site for someone, read a page through its structure and often a screenshot as well. An overlay that covers the page changes both. In the screenshot, the content is hidden or dimmed. In the page structure, a modal can make everything behind it unreachable until it's closed. So the agent's first job is to get rid of it. That goes wrong in a few predictable ways: - **No button it can recognize.** A close control drawn as an icon with no text or label has no name, so the agent can't tell it apart from any other shape on the screen. - **A fake button.** A styled `<div>` or `<span>` that responds to a mouse click but isn't a real button. Many agents won't recognize it as something they can press. - **Unclear choices.** "Customize", "Learn more" and "OK" side by side, or a reject option hidden behind a settings screen. The agent may pick the wrong one or stall. - **Overlays that come back.** A pop-up that reappears on every page, or a second pop-up after the first is closed, costs the agent a step every time. - **Something sitting on the main button.** A chat widget or sticky banner placed over "Add to cart" means the agent's click lands on the wrong thing. For the business, this is a quiet loss. The agent doesn't complain or fill in a feedback form. It tells the person it couldn't get through, or moves on to a site where it could. ### What AgentScore checks The [AgentScore](https://ghostagentlab.com/agentscore/) check is called **"Pop-ups and banners can be dismissed by agents"**, and it's one of the higher-weighted checks in the Navigability category. It works like this: 1. It opens your home page in a real browser at desktop size and waits for it to load. 2. It looks for anything fixed to the screen (a pop-up, a banner, a drawer) that covers 15% or more of the visible window. A slim header bar along the top doesn't count. 3. If it finds one, it reads the buttons and links inside it and looks for one with a clear dismiss label, such as "Accept", "Reject", "Decline", "Close", "Got it", "No thanks", "Continue" or a labelled ×. | What AgentScore finds | Result | | --- | --- | | Nothing covers the page on arrival | Pass | | An overlay covers the page, but it has a clearly labelled button to close or answer it | Warning: agents can get past it, but every one has to deal with it first | | An overlay covers the page and has no clearly labelled way out | Fail | A second check, **"Agents can find and press your add-to-cart or sign-up button"**, catches the other half of the problem. It finds the main button on a product or pricing page and tests whether something else, usually a pop-up, banner or chat widget, sits on top of it. If it does, the check fails, because an agent's click would land on the overlay instead. > **This isn't about skipping consent.** Where the law requires you to ask for cookie consent, keep asking. The goal is a banner that an AI agent acting for a person can answer as easily as the person could, not one that disappears. Talk to whoever owns privacy before changing what your banner asks or when. ### The usual suspects | Overlay | What tends to go wrong | Better approach | | --- | --- | --- | | Cookie consent banner | Full-screen modal; icon-only close; reject hidden in settings | A bar at the bottom of the screen with "Accept all" and "Reject all" as real, named buttons side by side | | Newsletter or discount pop-up | Appears on arrival, covers the page, close button is a small unlabelled icon | Wait until someone has engaged; use a labelled "Close" or "No thanks"; don't show it on product or checkout pages | | Country or region selector | Blocks the page until a choice is made, often with a custom dropdown | A small, dismissible suggestion ("Shopping from Canada? Go to the Canadian store") and locale in the URL | | Age check | Custom date pickers or a fake button | Where required, a short form with a native field and a real "Confirm" button | | Chat widget | Opens itself on load, or its launcher covers buttons on smaller screens | Start collapsed, keep the launcher small and clear of the main buttons | | App install or promo banner | Sticky and tall, pushing content down or covering it | Keep it short, dismissible with a named button, and remember the choice | ### What an agent-friendly banner looks like Here's a cookie banner that people, screen readers and AI agents can all handle. It sits at the bottom of the page, doesn't block what's behind it, and every choice is a real button with a plain name. Traps agents - Covers the whole page until it’s answered - The × has no name an agent can read - Rejecting is hidden behind “Settings” Works for agents - Sits at the edge, so the page stays usable - Every choice is a real button with words - Rejecting takes one step, like accepting *Two cookie banners. The first stops every agent until it’s answered and gives it nothing it can name. The second can be answered in one step, or left alone.* ``` <div class="consent-bar" role="region" aria-label="Cookie choices"> <p>We use cookies to run the store and, with your OK, to measure ads. <a href="/pages/cookies">Cookie policy</a></p> <button type="button">Reject all</button> <button type="button">Accept all</button> <button type="button">Choose cookies</button> </div> ``` And the most common failure, the icon-only close button, fixed in one attribute: ``` <!-- An agent sees: button, no name --> <button class="modal-x"><svg>…</svg></button> <!-- An agent sees: button "Close" --> <button class="modal-x" aria-label="Close"><svg aria-hidden="true">…</svg></button> ``` A few rules that make the difference: - **Use real buttons with words.** "Accept all", "Reject all", "Close", "No thanks". If the design calls for an icon, give it an `aria-label`. - **Keep it to the edge.** A bar along the bottom leaves the page usable. A centered modal over a dimmed page forces every visitor, human or agent, to deal with it first. - **Make rejecting as easy as accepting.** Several European privacy regulators have said this about cookie banners, and it also means an agent told to decline tracking can do so in one step. - **Let Escape close it** where closing is allowed, and return focus to the page afterwards. - **Remember the answer.** Once a choice is made, don't ask again on the next page. - **One overlay at a time.** Don't stack a discount pop-up on top of a cookie banner on the first page view. ### Pop-ups and the pages that earn money The home page is where AgentScore looks for overlays, but the pages that matter most are product pages, the cart and checkout. That's where a covered button costs a sale. Check those pages on a narrower screen too: a chat launcher that sits harmlessly in a corner on a laptop can land on top of "Add to cart" on a phone-sized window. If marketing relies on an email capture pop-up, that's a reasonable trade-off to discuss rather than a rule to break. Showing it after someone has scrolled or spent time on the page, rather than on arrival, keeps most of its value for people and removes it from the first moment an agent arrives. ### How to test it 1. **Open the site in a private window.** That's what a first-time visitor and most agents see: no cookies, no remembered choices. 2. **Try to get past every overlay with the keyboard alone.** Tab to the buttons and press Enter. If you can't reach the close button, or can't tell which control has focus, an agent will struggle too. 3. **Check the names.** In Chrome's developer tools, the Accessibility pane shows each button's name. An empty name is a problem. 4. **Visit a product page and the cart** at a narrow window size and make sure nothing covers the main button. 5. **Run [AgentScore](https://ghostagentlab.com/agentscore/).** It reports any overlay it found, how much of the screen it covered and which buttons it offered, and checks whether your add-to-cart or sign-up button is covered. Overlays are one part of how agents find their way around a page. For the rest, see [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/), and for the other thing that often greets agents on arrival, a CAPTCHA or bot challenge, see [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/). To see how agents handle the whole journey, not just the first screen, read [how to test your key journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). ### Sources and further reading - [Dialog (modal) pattern (ARIA Authoring Practices Guide)](https://www.w3.org/WAI/ARIA/apg/patterns/dialog-modal/) (W3C) - [Report of the work undertaken by the Cookie Banner Taskforce](https://www.edpb.europa.eu/documents/task-force-report/report-of-the-work-undertaken-by-the-cookie-banner-taskforce_en) (European Data Protection Board) --- ## Link your key pages from the home page > AI agents start at your home page and follow links. Make sure products, the cart and pricing are linked in plain HTML so every agent can find them. Published October 9, 2026 by Ghost Agent Labs · Navigability · https://ghostagentlab.com/articles/link-key-pages-home-page/ ### Key takeaways - AI agents usually start at your home page and follow its links, so the pages that make you money should be one link away. - Menus built by JavaScript, buttons posing as links and vague link text can hide products, cart and pricing from agents even when people see them. - View your home page's source and search for links to your cart, pricing or products; if they're missing, ask your developer to add them. When an AI agent is asked about your business, it usually starts where people do: your home page. From there it follows links. If the links to your products, cart or pricing aren't in the page's HTML, many agents never find those pages, however good they are. It's one of the simplest things to get right, and one of the easiest to break without noticing. ### How agents find their way from the home page An AI assistant answering "how much is the Pro plan at northwind.example?" or an agent asked to "buy two bags of the house blend" needs to get from your home page to the right page quickly. Agents differ in how they do that: - **Agents that read the HTML.** Many AI assistants and AI search crawlers fetch the page as your server sends it and read the links in it. They don't run JavaScript, so a menu that's built in the browser is invisible to them. (See [why AI agents can't see your JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/).) - **Browser agents.** These open a real browser and click around. They can see menus built with JavaScript, but they still need real links and clear names to choose from, and every extra click is a chance to get lost. Either way, the home page is the map. If the pages that make you money aren't on it, the agent has to guess, search, or fall back on your sitemap, if it looks for one at all. ### What AgentScore checks [AgentScore](https://ghostagentlab.com/agentscore/) goes looking for your key pages the way an agent would, in this order: 1. **Links in the home page's HTML.** It reads every link to your own site in the page as your server sends it, before any JavaScript runs. 2. **Your sitemap,** if nothing suitable was linked. 3. **A couple of common addresses,** such as `/pricing` or `/cart`, for anything still missing. It recognizes a key page by its address or its link text. Addresses like `/products/house-blend`, `/pricing`, `/plans` and `/cart` count, and so do links labelled "Pricing", "Plans", "Cart", "Basket" or "Bag". The check, **"Key pages are linked from the home page"**, then asks one question: did the home page link to them directly? | Type of site | Passes when the home page HTML links to | | --- | --- | | Online store | A product page or the cart | | Software or subscription business | The pricing page | | Neither found anywhere | Not tested | If AgentScore only found your key pages through the sitemap or a common address, the check gives a warning: the pages exist, but an agent reading your home page wouldn't have been led to them. The same pages are then used for other checks, such as whether AI agents can open them and whether prices are readable, so linking them clearly helps more than one result. ### Why links go missing Most sites that get a warning here do have links to their products and pricing. People can see them. The links just aren't where agents look. - **Menus built by JavaScript.** A mega menu or mobile menu that's fetched and drawn after the page loads isn't in the HTML at all. - **Buttons pretending to be links.** A "Shop now" that's a `<button>` or `<div>` with a click handler goes nowhere for an agent that reads links. So does `href="#"` with a script. - **Vague link text and addresses.** "Discover", "Explore the range" or "See what it costs", pointing to `/landing/q4-campaign`, gives neither people nor agents much to go on. - **Product links only in a carousel** that's loaded after the page renders, or only inside a search box. - **A cart icon with no link.** A cart drawer that opens with a script, with no link to a cart page behind it. ### What a good home page links to Think of the questions people ask AI assistants about businesses like yours, and make sure the answer is one link away from the home page. | Type of site | Link to from the home page | | --- | --- | | Online store | Main categories, a few bestselling products, the cart, shipping, returns, contact | | Software company | Pricing, sign-up, product overview, docs, contact sales | | Service business | Services and prices, booking, locations and hours, contact | Footer links count. A plain footer with "Pricing", "Shipping" and "Returns" is often the most reliable route an agent has, because it's on every page and rarely depends on JavaScript. ### What the HTML should look like The fix is ordinary links, present in the HTML your server sends, with clear text: ``` <!-- Hard for agents: no link in the HTML, vague text --> <div class="nav-item" onclick="openMenu('shop')">Discover</div> <span class="cart-icon" onclick="openCart()"></span> <!-- Easy for agents: real links, plain names --> <nav aria-label="Main"> <a href="/collections/coffee">Coffee</a> <a href="/collections/equipment">Brewing equipment</a> <a href="/pricing">Pricing</a> <a href="/cart" aria-label="Cart"><svg aria-hidden="true">…</svg></a> </nav> ``` - **Use `<a href>` with a real address** for anything that goes to another page. A cart drawer is fine, as long as the cart icon is also a link to a cart page. - **Render the main menu on the server.** If your platform or theme builds the menu in the browser, ask your developer to output at least the top-level links in the HTML. - **Name links for what's behind them.** "Pricing" beats "See what it costs"; "Shop coffee" beats "Discover". - **Keep addresses readable.** `/pricing`, `/cart` and `/products/house-blend` tell an agent what a page is before it opens it. - **Make menus open on click, not only on hover,** and make the top-level items real links too, so there's always a route in. ### How to check your own home page 1. **View the source.** In most browsers, right-click the page and choose "View page source" (not "Inspect", which shows the page after JavaScript has run). Search for `href="/cart"`, `/pricing` or `/products/`. If they're not there, agents that read HTML can't see them. 2. **Turn JavaScript off** in your browser's settings and reload the home page. Whatever navigation is left is what those agents get. 3. **Click through with the keyboard.** Tab from the top of the page to a product and the cart. If you can't, browser agents may not either. 4. **Run [AgentScore](https://ghostagentlab.com/agentscore/).** It tells you whether your products, cart or pricing were linked from the home page, or only turned up another way, such as through the sitemap. > **Your sitemap is a safety net, not a substitute.** A good [XML sitemap](https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/) helps crawlers find every page, and [llms.txt](https://ghostagentlab.com/articles/llms-txt/) can point AI straight to your most important ones. But plenty of agents never read either. Links on the home page are the route every agent can follow. Once agents can reach your key pages, the next questions are whether they can read them and act on them. See [prices AI agents can read](https://ghostagentlab.com/articles/machine-readable-prices/), [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/) and [site search and filters AI agents can use](https://ghostagentlab.com/articles/site-search-filters-ai-agents/). ### Sources and further reading - [SEO link best practices for Google](https://developers.google.com/search/docs/crawling-indexing/links-crawlable) (Google Search Central) --- ## Site search and filters AI agents can use > Build a search box, results URLs and filters that AI agents can operate, so they find the right products instead of giving up or guessing. Published October 9, 2026 by Ghost Agent Labs · Navigability · https://ghostagentlab.com/articles/site-search-filters-ai-agents/ ### Key takeaways - AI agents shop like people do, searching then filtering, so a search box and filters they can operate decide whether they find your products. - Unlabelled search icons, results drawn only by scripts and custom filter widgets make agents give up or pick the wrong product. - Give every search and filtered view its own readable web address that shows the same results when opened fresh; it helps people too. Ask an AI agent to "find a waterproof trail shoe under $150 in size 10" and, on most stores, it does what a person would: types into the search box, then narrows the results with filters. If the search box is hard to find, the results only exist inside a script, or the filters are custom widgets it can't operate, the agent gives up or picks the wrong thing. This guide covers how to build search and filters that AI agents can use, and that work better for people too. ### How agents use search and filters There are two broad ways an AI agent gets to a list of products: - **Through the page.** Browser agents open your site, find the search box by its label, type, press Enter, and then click filters and sort options, much as a keyboard or screen-reader user would. - **Through the address.** If your search and filtered results have their own URLs, an agent can go straight to `/search?q=trail+shoes` or a filtered category page. AI assistants that fetch pages rather than click through them can only work this way: they can't type into a box, but they can open a link. A site that supports both gives every kind of agent a route to the right products. A site that supports neither forces agents to browse category pages one by one, which is slow and often ends with a worse answer. ### A search box agents can find and use The search box should be a real form that works without anything clever. That means a labelled field, a named submit button, and a results page that loads when the form is submitted. ``` <form role="search" action="/search" method="get"> <label for="q">Search products</label> <input id="q" name="q" type="search" autocomplete="off"> <button type="submit">Search</button> </form> ``` - **Label it.** A visible label or an `aria-label` such as "Search products". A magnifying-glass icon on its own has no name an agent can read. - **Keep it visible.** A search box that only appears after clicking an unlabelled icon is an extra step agents often miss. If space is tight, make the icon a real button named "Search". - **Make Enter work.** Pressing Enter should submit the search and open a results page. Predictive suggestions as you type are fine, but they shouldn't be the only way to get results. - **Accept the words people use.** Product names, product types, brands, SKUs, common misspellings and synonyms ("sneakers" and "trainers"). An agent passes on what the person asked for, in their words. ### Give results and filters their own URLs This is the single most useful change for AI agents, and it's good for people too: results can be bookmarked, shared and reached with the back button. Every search and every combination of filters should be reflected in the address, using readable parameters: ``` https://northwind.example/search?q=trail+shoes https://northwind.example/collections/shoes?waterproof=yes&size=10&max_price=150 https://northwind.example/collections/shoes?sort=price-asc&page=2 ``` - **Update the URL when filters change.** If your filters work without reloading the page, update the address as they're applied, so the current view always has a link. - **Make the URL load the same results.** Opening that link fresh, with no history, should show the same filtered list. - **Put results in the HTML.** If results are only drawn by JavaScript, agents that read HTML see an empty page. See [why AI agents can't see your JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). - **Use plain parameter names and values.** `size=10` and `color=black` are guessable. `f=a8x2k` isn't. ### Filters as real controls Filters are where many stores lose agents. They're often built as custom widgets that look good but aren't recognizable as checkboxes, options or buttons. The fix is to use the controls the browser already provides. | Filter | Hard for agents | Easy for agents | | --- | --- | --- | | Size, brand, feature | Clickable boxes made from `<div>`s | Checkboxes with labels, grouped in a `<fieldset>` with a `<legend>` | | Color | Swatches with no text, only a background color | Checkboxes or buttons named "Black", "Navy", with the swatch as decoration | | Price | A drag-only slider | "Min" and "Max" number fields, with or without a slider alongside | | Sort order | A custom dropdown that opens on hover | A native `<select>`, or a list of real links | | More results | Infinite scroll only | Numbered page links, or a "Load more" button backed by a `?page=2` link | ``` <fieldset> <legend>Size</legend> <label><input type="checkbox" name="size" value="9"> 9 (14)</label> <label><input type="checkbox" name="size" value="10"> 10 (22)</label> </fieldset> <label for="sort">Sort by</label> <select id="sort" name="sort"> <option value="relevance">Most relevant</option> <option value="price-asc">Price: low to high</option> </select> ``` A few more details that save agents from guessing: - **Say how many results there are** in text ("24 products"), and update it when filters change. An `aria-live` region lets assistive tech, and agents reading the page structure, notice the change. - **Show active filters** as named buttons that remove them ("Remove filter: Size 10"), plus a "Clear all". - **Handle no results helpfully.** Say plainly that nothing matched, and offer a way out: related categories, or the same search without the last filter. - **Keep filter panels reachable.** On smaller screens, the "Filters" toggle should be a real, named button, not an icon. ### Tell agents how your search works Once your search has a predictable URL, you can describe it so agents don't have to discover it: - **In llms.txt.** A line such as "Search: `https://northwind.example/search?q={query}`" in your [llms.txt file](https://ghostagentlab.com/articles/llms-txt/) hands the pattern to any AI that reads it. - **In structured data.** schema.org lets a `WebSite` describe its search with a [SearchAction](https://schema.org/SearchAction). Google retired the search box it used to show for this in 2024, so don't expect a search result feature from it, but it remains a standard, machine-readable description of your search URL. - **As a tool.** The most direct option is to let agents call your search as a tool, through an MCP server or WebMCP in the browser. Both are newer and still settling; see [MCP and WebMCP for websites](https://ghostagentlab.com/articles/mcp-webmcp-for-websites/). > **Watch your robots.txt.** Many sites stop search engines indexing internal search results, which is sensible. But a blanket `Disallow: /search` for every bot also tells AI assistants that respect robots.txt not to open those pages when a person asks them to. Decide deliberately which agents that rule should apply to. See [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). ### What AgentScore and Ghost Agents test AgentScore doesn't have a separate check for search, but several of its checks cover the building blocks. On the home page, where most search boxes live, it checks that form fields are labelled, that buttons and links have names, and that clickable things are real buttons and links. "Content loads without JavaScript" catches a home page whose content only appears once scripts run, a sign that results and filters may be built the same way. The free scan doesn't run a shopping journey itself; a [Ghost Agent](https://ghostagentlab.com/ghost-agent/) test does, and a search box or filter that doesn't work shows up in its step-by-step replay. Search is also one of the journeys [Ghost Agents](https://ghostagentlab.com/ghost-agent/) can test for you. You choose the journey, such as "search for a waterproof trail shoe, filter to size 10 and open the first result", and a Ghost Agent works through it like a customer would, on the schedule you set. When a theme update or a new search app breaks it, you find out before customers do. See [how to test your key journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). ### A quick checklist 1. Search box is visible, labelled, and submits with Enter to a results page. 2. Searches and filtered views have their own readable URLs that load the same results when opened fresh. 3. Results are in the page HTML, not only drawn by JavaScript. 4. Filters are checkboxes, selects, number fields and buttons with names, not styled boxes. 5. Result counts, active filters and "no results" are shown in text. 6. Pagination uses real links. 7. Your search URL is described in llms.txt, and robots.txt doesn't shut AI assistants out of it by accident. For the controls on the rest of the page, see [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). For getting agents to your categories in the first place, see [link your key pages from the home page](https://ghostagentlab.com/articles/link-key-pages-home-page/). ### Sources and further reading - [SearchAction](https://schema.org/SearchAction) (schema.org) - [Grouping controls (forms tutorial)](https://www.w3.org/WAI/tutorials/forms/grouping/) (W3C) - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) (IETF) --- ## Size, color and variant pickers AI agents can use > Build size and color pickers AI agents can use: named options, a readable selected state, clear sold-out options, variant URLs and prices that update. Published October 9, 2026 by Ghost Agent Labs · Navigability · https://ghostagentlab.com/articles/variant-pickers-ai-agents/ ### Key takeaways - Choosing a size or color is where many product pages lose AI agents, leading to abandoned tasks or wrong items in the cart. - Agents can't read color swatches with no name, a selection shown only by a border, or sold-out sizes that are merely faded. - Ask your developer or theme provider to rebuild swatches as named radio buttons styled the same way, the simplest and most reliable picker. Ask an AI agent to buy "the navy jacket in a medium" and the hard part isn't finding the jacket. It's choosing navy and medium. Size, color and other variant pickers are where many product pages lose agents: the swatch has no name, the selected size looks different but reads the same, and the sold-out option can still be clicked. This guide covers how to build variant pickers AI agents can use without guessing. ### Why variant pickers trip agents up A person sees a row of colored circles, notices a thicker border around one, and sees a crossed-out "XL". A browser agent mostly reads the page's structure: each control's role, its name and its state. If the swatch is a `<div>` with a background color, it has no role and no name. If "selected" is shown only with a border, there's no state to read. The agent may click the right circle by chance, click the wrong one, or not realize there's a choice to make at all. The cost is direct. An agent that can't choose a size either gives up and tells the shopper it couldn't complete the task, or adds the wrong item to the cart. Both are lost or returned orders. And the same problems affect shoppers who use a keyboard or a screen reader, so fixing them helps people too. Agents need four things from a picker: what each option is called, which one is chosen, which ones can't be bought, and confirmation that the page changed after they chose. ### 1. Make each option a real control with a name The simplest, most reliable picker is a group of native radio buttons. Only one option can be chosen, the browser handles the keyboard, and every agent knows what a radio button is. You can style the label to look like a swatch or a size box. ``` <fieldset> <legend>Color</legend> <label class="swatch"> <input type="radio" name="color" value="navy" checked> <span class="dot" style="background:#1f2a44" aria-hidden="true"></span> Navy </label> <label class="swatch"> <input type="radio" name="color" value="olive"> <span class="dot" style="background:#5b5e3a" aria-hidden="true"></span> Olive </label> </fieldset> ``` - **Name every option.** The color name, such as "Navy", should be the control's name, as visible text or at least an `aria-label`. A hex code or an image file name isn't a name. - **Name the group.** A `<fieldset>` with a `<legend>` ("Color", "Size") tells the agent what the choice is about. - **Use the words shoppers use.** "Medium" or "M" is fine. An internal code like "SZ-03" isn't. If sizes need context ("UK 8 / US 4"), put it in the name. - **Keep swatch images as decoration.** If a swatch is a photo of the fabric, give the control the color name and the image empty `alt=""`, so the name isn't read twice. If your design uses buttons rather than radios, that works too, as long as they are real `<button>` elements with names. AgentScore's checks that buttons and links have names, and that clickable things are real buttons and links, look for exactly these problems, though they run on your home page. A product page needs the same care. ### 2. Show which option is selected, in the code Selection is the detail most often shown only visually. Radio buttons report it for free through `checked`. Custom buttons have to say it themselves: ``` <div role="group" aria-labelledby="size-label"> <span id="size-label">Size</span> <button type="button" aria-pressed="false">S</button> <button type="button" aria-pressed="true">M</button> <button type="button" aria-pressed="false">L</button> </div> ``` - Use `aria-pressed="true"` on the chosen button and `"false"` on the others, and update them when the choice changes. - Alternatively, use `role="radio"` with `aria-checked` inside a `role="radiogroup"`. It's more work to get the keyboard right, which is why native radios are usually the better choice. - Repeat the current choice in text near the picker ("Color: Navy"). It gives agents and people a plain confirmation. ### 3. Mark out-of-stock options clearly Hiding a sold-out size makes the shopper wonder whether it ever existed. Showing it with only a strike-through or faded color leaves an agent free to choose it, then fail at "Add to cart". Say it in words and in the control's state. | Approach | What an agent sees | | --- | --- | | Option faded or struck through with CSS only | A normal option it can pick | | Option removed from the page | No sign the size exists | | Native radio with `disabled` and "XL, sold out" in the label | An option that can't be chosen, and why | | Button with `aria-disabled="true"` and "XL, sold out" in its name | The same, and it stays reachable by keyboard | If you offer back-in-stock alerts, make that a separate, named button ("Email me when XL is back"), so an agent can tell it apart from buying. ### 4. Give each variant its own URL Many AI assistants fetch pages rather than click through them. They can't press a swatch, but they can open a link. If each variant has its own address, an assistant can point a shopper straight to "the navy jacket in medium", and a browser agent can recover if a click doesn't take. ``` https://northwind.example/products/field-jacket?color=navy&size=m https://northwind.example/products/field-jacket?variant=40318 ``` - Update the address when a shopper picks an option, so the current view always has a link. - Make the URL load with that variant already selected, its price and stock shown, when opened fresh. - Point the canonical URL at the main product page unless variants are really separate products. See [canonical URLs and duplicate pages](https://ghostagentlab.com/articles/canonical-urls-ai-agents/). ### 5. Update price, stock and photos where agents can see it Choosing a larger size or a different material often changes the price. If the new price only appears in an image, or changes quietly in a corner of the page, an agent may report the old one to the shopper. - Show price and availability as text, and update them when the variant changes. - Wrap the price and stock line in a live region (`aria-live="polite"`), so the change is announced to assistive tech and is easier for agents reading the page structure to notice. - Keep "Add to cart" a real button with the same name throughout. If it's disabled until a size is chosen, say so next to it: "Choose a size". AgentScore's add-to-cart check accepts a button that's disabled until options are chosen, because agents can handle that, as long as they can tell what's missing. ### 6. Describe variants in your structured data Agents that read your page HTML rather than click it rely on structured data for the full list of options. schema.org has a [ProductGroup](https://schema.org/ProductGroup) type for this: one group, with `variesBy` naming the dimensions and `hasVariant` listing each variant as a `Product` with its own offer. Google documents the pattern in its [product variant structured data guide](https://developers.google.com/search/docs/appearance/structured-data/product-variants). ``` { "@context": "https://schema.org", "@type": "ProductGroup", "name": "Field jacket", "productGroupID": "FJ-100", "variesBy": ["https://schema.org/color", "https://schema.org/size"], "hasVariant": [{ "@type": "Product", "sku": "FJ-100-NAV-M", "color": "Navy", "size": "M", "url": "https://northwind.example/products/field-jacket?variant=40318", "offers": { "@type": "Offer", "price": "189.00", "priceCurrency": "USD", "availability": "https://schema.org/InStock" } }] } ``` AgentScore's "Product pages give price and stock in a form agents can read" check reads offers inside `ProductGroup` and `hasVariant`, as well as on a plain `Product`. For more on the fields that matter, see [product data AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/) and [prices AI agents can read](https://ghostagentlab.com/articles/machine-readable-prices/). ### How to test your pickers 1. **Use only the keyboard.** Tab to the color and size pickers, choose an option, and add to cart. If you can't, many agents can't either. 2. **Inspect the accessibility tree.** In Chrome's developer tools, the Accessibility pane should show each option as a radio or button with a name like "Navy", and its checked or pressed state. 3. **Pick a sold-out size.** The page should tell you, in words, before you reach the cart. 4. **Copy the URL after choosing.** Open it in a private window. The same variant should be selected. 5. **Run a journey.** AgentScore's free scan doesn't choose options on your product pages, so the best test is an agent trying it. A [Ghost Agent](https://ghostagentlab.com/ghost-agent/) can run "find the field jacket, choose navy in medium, add it to the cart" on a schedule and replay each step when it gets stuck. See [testing key journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). > **Where to start.** If your theme uses `<div>` swatches, switching to labelled radio buttons styled the same way usually fixes naming, selection state and keyboard use in one change. Ask your developer or theme provider for that first. For the rest of the page's controls, see [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). For what happens after "Add to cart", see [carts AI agents can manage](https://ghostagentlab.com/articles/cart-ai-agents/). ### Sources and further reading - [Radio group pattern (ARIA Authoring Practices Guide)](https://www.w3.org/WAI/ARIA/apg/patterns/radio/) (W3C) - [Button pattern (ARIA Authoring Practices Guide)](https://www.w3.org/WAI/ARIA/apg/patterns/button/) (W3C) - [ProductGroup](https://schema.org/ProductGroup) (schema.org) - [Product variant structured data](https://developers.google.com/search/docs/appearance/structured-data/product-variants) (Google Search Central) --- ## Infinite scroll, “load more” and pagination for AI agents > Infinite scroll can hide most of your catalog from AI agents. Back it with real paginated URLs, a “load more” link and a sitemap. Published October 9, 2026 by Ghost Agent Labs · Navigability · https://ghostagentlab.com/articles/infinite-scroll-pagination-ai-agents/ ### Key takeaways - With infinite scroll alone, many AI agents see only the first batch of products, so a large category can look like a small one. - If a shopper asks for a product that only loads further down, an assistant may tell them you don't sell it. - Keep infinite scroll or "load more" if you like it, but give each page of results its own web address and a real link. Infinite scroll feels effortless to a shopper with a thumb. To many AI agents, it's a wall. Products that only load as someone scrolls may never be seen by an assistant that fetches the page, and a browser agent can waste its time scrolling and waiting. The fix isn't to drop infinite scroll or "load more". It's to put real, linked pages underneath them. ### What agents see on a long list Different agents meet a category page in different ways: - **Assistants that fetch pages**, such as ChatGPT or Perplexity looking something up for a person, usually read the HTML your server sends. They don't scroll, and often don't run JavaScript. They see the first batch of products and nothing more. - **AI search crawlers** also read HTML and follow links. If page two has no link, it doesn't exist for them. - **Browser agents** can scroll and click, but each scroll is a step that takes time. They can't always tell whether more products are coming or the list has ended. So a category of 300 products that shows 24 at a time, with more loading on scroll, can look like a category of 24. If the product a shopper asked for is number 61, the assistant may tell them you don't sell it. ### Give every page of results its own URL The foundation is the same for infinite scroll, "load more" and numbered pages: each chunk of results should exist at its own address, and that address should load those results when opened fresh. ``` https://northwind.example/collections/coffee https://northwind.example/collections/coffee?page=2 https://northwind.example/collections/coffee?page=3 ``` - **Use query parameters or paths, not fragments.** `?page=2` and `/page/2` are separate URLs. `#page=2` is not: servers never see what follows the `#`, so it can't return different results. - **Render the results in the HTML.** `?page=2` should arrive with its products already in it, not an empty shell that fills in later. See [why AI agents can't see your JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). - **Keep the order stable.** Page two should show the same products whether you arrive from page one or open it directly. Random or personalized ordering makes pages unreliable to follow. - **Carry filters and sort along.** `?roast=dark&sort=price-asc&page=2` should keep both. Our guide to [site search and filters AI agents can use](https://ghostagentlab.com/articles/site-search-filters-ai-agents/) covers filter URLs in detail. Google's guidance on [pagination and incremental page loading](https://developers.google.com/search/docs/specialty/ecommerce/pagination-and-incremental-page-loading) takes the same approach, and it works for AI agents for the same reasons. ### Link the pages together with real links Unique URLs only help if agents can find them. Each page should link to the next one, and ideally to the previous one and a few page numbers, with ordinary `<a href>` links in the HTML. ``` <nav aria-label="Pagination"> <a href="/collections/coffee?page=1">Previous page</a> <a href="/collections/coffee?page=1">1</a> <a href="/collections/coffee?page=2" aria-current="page">2</a> <a href="/collections/coffee?page=3">3</a> <a href="/collections/coffee?page=3">Next page</a> </nav> ``` - Wrap the links in `<nav>` with a name like "Pagination", so agents can tell it apart from the main menu. - Name arrow links in words ("Next page"), not just "›". - Mark the current page with `aria-current="page"`. - Say how many products there are in text, such as "Showing 25–48 of 312". It tells an agent there's more to see. ### Make "load more" a button, backed by a link "Load more" is a good middle ground for shoppers, and it can work well for agents if it's built as two things at once: a real button for people and browser agents, and a real link for everyone else. ``` <a class="load-more" href="/collections/coffee?page=3"> Load more products (264 remaining) </a> ``` Start with a plain link to the next page. Then let JavaScript take over: intercept the click, fetch the next page, append the products, and update the address with `history.pushState` so the URL matches what's on screen. If the script doesn't run, or the agent doesn't run scripts, the link still goes to page three. If you prefer a `<button>`, keep the numbered page links alongside it so there's still a route. - Give the button a specific name: "Load more products", not "More" or an arrow icon. - Announce new results with a live region ("24 more products loaded") so the change is easier for agents and assistive tech to notice. - Move keyboard focus to the first new product, or leave it on the button, but don't send it back to the top of the page. ### Infinite scroll: keep it, but add pages underneath If you want true infinite scroll, treat it as an enhancement over paginated pages. As each new batch loads, update the URL to the page number the shopper has reached. Keep the paginated links in the HTML, even if they're visually tucked away at the bottom. And make sure the footer is reachable: if the page grows every time someone nears the bottom, neither people nor agents can reach your contact, shipping and returns links. | Pattern | Fetching assistants and crawlers | Browser agents | | --- | --- | --- | | Infinite scroll only | See the first batch only | Must scroll and wait, can't tell when it ends | | "Load more" button with no URL | See the first batch only | Can click, if the button has a name | | "Load more" link to `?page=2` | Can follow every page | Can click or follow | | Numbered page links | Can follow every page | Can jump straight to a page | ### What about rel="next" and rel="prev"? For years, `<link rel="next">` and `rel="prev"` in the page head were the standard way to tell search engines that pages belonged in a series. Google said in 2019 that it no longer uses them for indexing. Other search engines and tools may still read them, and they're part of HTML, so they do no harm. But don't rely on them: an agent following links needs the visible `<a href>` links in the page. Two related head tags matter more: - **Canonical.** Give each paginated page its own canonical URL. Pointing `?page=2` at page one tells crawlers to ignore the products on page two. See [canonical URLs and duplicate pages](https://ghostagentlab.com/articles/canonical-urls-ai-agents/). - **Robots meta.** Don't mark paginated pages `noindex` by default. Pages that stay out of the index can, over time, get less attention from crawlers, and so can the products linked from them. ### Back it up with a sitemap Even with perfect pagination, some products sit deep in a long list. Your XML sitemap should list every product page directly, so crawlers don't depend on paging through categories to find them. AgentScore's "Valid sitemap" check confirms you have one. See [XML sitemaps for AI agents](https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/). The sitemap is the safety net. The links are still the main route. ### How to check your category pages 1. **View the source** of a category page and search for a product name from further down the list. Then look for a link to page two. 2. **Open `?page=2` directly** in a private window. It should show the second set of products, not the first and not an empty page. 3. **Turn off JavaScript** and try to reach the last page of a category. 4. **Check the footer** is reachable on a category page with infinite scroll. 5. **Run AgentScore.** It has no pagination check, but "Content loads without JavaScript" and "Key pages are linked from the home page" show whether HTML-only agents can see your content and reach your products. > **Test it as a shopper would.** Ask a [Ghost Agent](https://ghostagentlab.com/ghost-agent/) to "find the cheapest dark roast in the coffee category" on a category of more than one page. The step-by-step replay shows whether it reached the later pages or settled for what loaded first. For the bigger picture of how agents move around a site, see [What is agent readiness?](https://ghostagentlab.com/articles/agent-readiness/) ### Sources and further reading - [Pagination best practices for Google](https://developers.google.com/search/docs/specialty/ecommerce/pagination-and-incremental-page-loading) (Google Search Central) - [WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/) (W3C) --- ## Error messages AI agents can understand and fix > Help AI agents recover from mistakes: error text tied to each field, live regions, useful 404 pages and HTTP status codes that match what the page says. Published October 9, 2026 by Ghost Agent Labs · Navigability · https://ghostagentlab.com/articles/error-messages-ai-agents/ ### Key takeaways - AI agents can only fix a form mistake if the error is written in text, says what to do, and sits beside the right field. - An unclear error can quietly end a journey, so you lose the order or sign-up without anyone telling you. - Start by submitting your checkout with a wrong ZIP code and checking whether the page text alone explains what to fix. Every journey hits an error eventually: a mistyped postcode, a promo code that's expired, a product that's gone. A person reads the red text, fixes it and carries on. An AI agent can only do the same if the error is written down, attached to the right field and reported honestly by your server. Otherwise it tries again, gives up, or tells the shopper something went wrong without saying what. ### Why errors matter more for agents An agent working for someone wants to finish the task. When a form doesn't go through, it needs to answer three questions: did something go wrong, which field caused it, and what should it enter instead? People answer those from visual cues, such as a red border, a shake, an icon or a message that flashes up and disappears. Agents mostly read the page's text and structure, and may not catch any of those cues. A clear error turns a failed attempt into a quick correction. An unclear one ends the journey, and you lose the order or sign-up without anyone telling you. The same fixes help people who use screen readers, and anyone filling in a form on a small screen. ### Write errors that say what to do Start with the words. An agent passes the message on, or acts on it, exactly as written. | Unhelpful | Helpful | | --- | --- | | Invalid input | Enter a 5-digit ZIP code, like 10001 | | Error | This email address is already registered. Log in or reset your password. | | Promo code not applied | The code AUTUMN10 expired on September 30 | | Something went wrong | We couldn't save your address. Please try again in a minute. | - Name the field and the problem, and give an example of what's accepted. - Say whether it's something the shopper can fix, or something on your side that may work later. - Don't rely on color, icons or a border alone. If it isn't in text, assume an agent won't see it. - Keep what was entered. Clearing the whole form after one mistake makes an agent start again, and some won't. ### Tie each message to its field An error floating near a form isn't enough. Agents, like screen readers, need to know which field it belongs to. Two attributes do that: ``` <label for="zip">ZIP code</label> <input id="zip" name="zip" autocomplete="postal-code" aria-invalid="true" aria-describedby="zip-error"> <p id="zip-error">Enter a 5-digit ZIP code, like 10001</p> ``` - `aria-invalid="true"` marks the field as having a problem. Remove it once the value is fixed. - `aria-describedby` links the field to its message, so the message is read as part of the field's description. - Put the message in the page as text next to the field, not in a tooltip that only appears on hover. - For required fields, use the `required` attribute, and the right `type` and `autocomplete` values, so agents can get it right first time. Labels matter here too: see [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). ### Summarize errors and announce changes On a long form, such as checkout or account sign-up, add a short summary at the top when it's submitted with problems. It lists each error as a link to its field, and gets keyboard focus so it's the first thing read. ``` <div role="alert" tabindex="-1" id="error-summary"> <h2>There are 2 problems with your details</h2> <ul> <li><a href="#zip">Enter a 5-digit ZIP code, like 10001</a></li> <li><a href="#phone">Enter a phone number with area code</a></li> </ul> </div> ``` For messages that appear without a page load, such as "Promo code applied" or "Only 2 left, quantity updated", use a live region. `role="alert"` is for urgent errors. `role="status"` or `aria-live="polite"` is for confirmations. Keep the message on screen until the next action: toasts that fade after three seconds are easy for an agent to miss. ### Make error pages say what happened Errors aren't only in forms. When an agent follows an old link or a product is discontinued, your error page is what it reads. It should explain, and offer a way forward. - **Not found pages** should say the page doesn't exist, and include a search box and links to main categories, so an agent can recover. - **Discontinued products** are better served by a page that says so, with links to alternatives, or a permanent redirect to the closest replacement. - **Out-of-stock products** should keep their page, show "Out of stock" in text, and set `availability` to `OutOfStock` in your product data. Don't turn them into errors. - **Server errors and maintenance** pages should say it's temporary and, if you can, when to try again. ### Send the status code that matches Agents and crawlers read your HTTP status code before they read a word of the page. If they disagree, the code usually wins. A "Page not found" message sent with a `200 OK` status, often called a soft 404, tells crawlers the page is fine, so the dead page may stay in AI search answers. A real product page that returns an error tells them it's gone. | Situation | Status code | | --- | --- | | Page loaded normally, including out-of-stock products | `200` | | Page moved for good | `301` or `308` to the new address | | Page doesn't exist | `404` | | Page removed on purpose and not coming back | `410` | | Too many requests | `429`, with a `Retry-After` header | | Something broke on your side | `500` | | Down for maintenance or overloaded | `503`, with a `Retry-After` header | Two common mistakes: redirecting every missing page to the home page, which leaves agents on a page that doesn't answer the question, and serving a block page with `200`, which looks like content. The meanings of these codes are defined in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110). For 429s and agents, see [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). For form submissions, what matters most is that the page sent back shows the errors in text, tied to their fields. Whether you re-render the form with `200` or use `422` is less important than making the message clear. ### What AgentScore and Ghost Agents show AgentScore doesn't have a check for error messages, since it never submits your forms. Several checks cover the groundwork. "Form fields are labelled" checks the home page's fields, and "Cart and checkout fields are labelled for agents" checks the cart and checkout. "AI agents can open your product, pricing and cart pages" compares the status codes AI assistants get with what a browser gets, which catches bot protection sending agents errors that people never see. To see what happens when an agent actually makes a mistake, run a journey. A [Ghost Agent](https://ghostagentlab.com/ghost-agent/) types the test details you give it and records each step, so if a form rejects them without saying why, the replay shows exactly where it stalled. Ghost Agents stop at the payment form, so payment errors are out of scope. To find the errors agents hit in real traffic, see [finding the errors AI agents hit in your logs](https://ghostagentlab.com/articles/agent-errors-in-logs/). ### A quick checklist 1. Every error message says what's wrong and how to fix it, in text. 2. Each field with an error has `aria-invalid="true"` and `aria-describedby` pointing to its message. 3. Long forms show an error summary that links to each field. 4. Messages that appear without a page load use a live region and stay on screen. 5. Entered values are kept after an error. 6. Missing pages return `404` or `410`, with a search box and useful links. 7. Out-of-stock products keep their page, with a `200` status. 8. Maintenance and rate-limit responses use `503` or `429` with `Retry-After`. > **Try it yourself.** Submit your checkout's address form with a wrong ZIP code, then read only the text on the page. If you can't tell which field is wrong and what it wants, neither can an agent. For more on the forms themselves, see [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/) and [sign-up, booking and lead forms AI agents can finish](https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/). ### Sources and further reading - [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html) (IETF) - [RFC 6585: Additional HTTP Status Codes](https://www.rfc-editor.org/rfc/rfc6585.html) (IETF) - [Understanding WCAG 2.2 Success Criterion 3.3.1: Error Identification](https://www.w3.org/WAI/WCAG22/Understanding/error-identification.html) (W3C) - [Accessible Rich Internet Applications (WAI-ARIA) 1.2](https://www.w3.org/TR/wai-aria-1.2/) (W3C) --- ## How to make checkout work for AI shopping agents > A step-by-step guide to the shopping journey for AI agents: finding products, choosing options, adding to cart, checkout and the payment handoff. Published October 8, 2026 by Ghost Agent Labs · Task completion · https://ghostagentlab.com/articles/agent-ready-checkout/ ### Key takeaways - Agents must find a product, choose options, add to cart and check out, and every step is a place a sale can be lost. - Custom size and color pickers are where agents most often fail on good sites, and forcing account creation stops many agents completely. - Walk your checkout from home page to payment using only the keyboard, and retest after every theme update or new app. When a person asks an AI assistant to buy something, the agent has to do everything a shopper does: find the product, choose the right options, add it to the cart, and get through checkout. Each step is a place it can get stuck, and a stuck agent is a lost sale. This guide walks the journey step by step. ### How agents shop today Agents buy in two ways, and you'll want to support both: - **Through your website.** A browser agent uses your site the way a person would, and typically hands control back to the person to sign in or pay. This works today on any site, as long as the agent can get through. - **Through an agentic commerce protocol.** The assistant completes the purchase through a direct integration, without driving your web pages. Examples include the Agentic Commerce Protocol used by ChatGPT and Google's Universal Commerce Protocol. Support usually comes through your commerce platform or payment provider. Protocols are growing fast, but for now most agents still shop through the website. That's the journey this guide covers. 1. Find the product Hover-only menus, and search that only works with scripts 2. Choose options Most common failure Custom size and color swatches with no names or roles 3. Add to cart A button covered by a banner, or no text confirming the add 4. The cart Quantity and remove buttons shown only as icons 5. Checkout Forced accounts, unlabelled fields and a CAPTCHA on every order 6. Payment Handed to the person The agent hands back to the person; a cart that empties during the handoff loses the sale *The six steps an agent works through for a shopper, and what most often stops it at each. Choosing options is where agents most often fail on otherwise good sites.* ### Step 1: Find the product - **Search works without JavaScript tricks.** A real search form with a labelled field and a submit button, whose results have their own URL (like `/search?q=decaf`), lets agents search directly. - **Categories are real links.** Category and product links should be ordinary `<a href>` links in the HTML, reachable without hover menus. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). - **Product pages describe the product.** Name, price, stock and key specifications in the HTML, with structured data. See [product data agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). ### Step 2: Choose options This is where agents most often fail on otherwise good sites. Size, color and quantity pickers are frequently custom-built swatches that look fine to people but have no names or roles agents can read. - Use native `<select>` elements or radio buttons, or give custom swatches a role and a name, for example a radio group with each option labelled "Size 10" or "Color: Navy". - Show unavailable options as disabled, with text saying why ("Out of stock"), not just a faded color. - Update price and stock text on the page when an option changes, so the agent can confirm it picked correctly. - Give each variant its own URL where possible (`?variant=…`), so an agent can link straight to the exact item. ### Step 3: Add to cart - The add-to-cart control is a real `<button>` with clear text, and nothing covers it, such as a cookie banner or chat widget. - Confirm the add in text: "Added to cart: Ethiopia Yirgacheffe, 340 g" with a link to the cart. An animation alone leaves the agent unsure whether it worked. - If a cart drawer slides out, make sure it has a labelled close button and a real link to the cart page. - Skip upsell pop-ups at this step, or make them easy to dismiss with a labelled button. ### Step 4: The cart - The cart lives at a stable URL, such as `/cart`, that agents can open directly. - Line items, quantities, subtotal and any discounts are shown as text. - Quantity controls and remove buttons have names ("Remove Ethiopia Yirgacheffe"), not just "×" or a trash icon. - The checkout button is a clearly named button or link, visible without scrolling past recommendations. ### Step 5: Checkout - **Offer guest checkout.** Forcing account creation is a dead end for many agents. Offer sign-in as an option, not a gate. - **Label every field and use autocomplete.** `autocomplete="shipping street-address"`, `postal-code`, `email` and `tel` tell an agent exactly where each detail goes. - **Make shipping options readable.** Each option's name, price and delivery estimate as text, with a real radio button to choose it. - **Write errors as text.** "Enter a 5-digit ZIP code", next to the field, rather than a red outline. - **Show the full total before payment**, including shipping and tax, so the agent can confirm it with the person before they pay. - **Avoid a CAPTCHA on every checkout.** Use risk-based checks that only challenge suspicious orders. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/). ### Step 6: Payment Most agents stop at payment and hand back to the person, and that's how it should be. Make the handoff smooth: a clear payment step, wallet options like Apple Pay, Google Pay or Shop Pay that a person can approve in one tap, and a cart that survives the handoff instead of emptying after a short timeout. To let assistants complete purchases on their own, ask your commerce platform or payment provider which agentic commerce protocols it supports. AgentScore checks whether your site advertises one. ### How to test your checkout 1. **Walk it with the keyboard only.** From the home page to the payment step, without a mouse. Anywhere you get stuck, an agent probably does too. 2. **Run [AgentScore](https://ghostagentlab.com/agentscore/).** For online stores, it checks that AI agents can open your product, cart and checkout pages, find and press the add-to-cart button, and fill in labelled checkout fields, and that there's a guest checkout. It doesn't run a full journey itself. 3. **Monitor it with [Ghost Agents](https://ghostagentlab.com/ghost-agent/).** They run the whole journey on a schedule with test details you provide, and alert you when it breaks. They stop when they reach a payment form, never enter card details, and refuse buttons that place orders, so testing never creates a real order. > **Retest after every change.** Checkout breaks for agents in the same ways it breaks for people: a theme update, a new app, a redesigned size picker. The difference is that agents don't email support. They just buy elsewhere. ### Sources and further reading - [Agentic Commerce Protocol](https://developers.openai.com/commerce) (OpenAI) - [Google Universal Commerce Protocol (UCP) Guide](https://developers.google.com/merchant/ucp) (Google for Developers) - [HTML Standard: Form control infrastructure (autofill)](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html) (WHATWG) --- ## Sign-up, booking and lead forms AI agents can finish > How to build sign-up, booking and quote forms AI agents can complete, and how to handle email verification, CAPTCHAs, date pickers and the handoff. Published October 9, 2026 by Ghost Agent Labs · Task completion · https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/ ### Key takeaways - When an assistant cannot finish your sign-up, booking or quote form, the lead goes to a competitor whose form it can. - A CAPTCHA on every submission stops AI agents as surely as spam bots, so prefer checks that only challenge suspicious activity. - Save the booking or lead before asking for email verification, so the work is not lost if the agent hands back to the person. "Book me a table for four on Friday." "Get me a quote for a new roof." "Sign me up for the free trial." More of these requests now go to an AI assistant instead of a search box, and the assistant has to finish the form on your site. If it can't, the lead goes to a competitor whose form it can. This guide covers sign-up, booking and lead forms, and the steps where agents most often get stuck. ### Why forms are where agents give up Reading a page is easy for an AI agent. Finishing a form is not. Every field has to be matched to something the person asked for, every choice has to be made, and every step has to confirm that it worked. Forms also tend to carry the things built to stop bots: CAPTCHAs, email verification and one-time codes. None of this means removing your defenses. It means designing the form so an agent can do the routine part, and handing over to the person at the moments that genuinely need them. ### The basics every form needs - **A visible label on every field.** Agents match "my phone number" to a field called "Phone number". Placeholder text alone is a weak substitute. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). - **The right field types and autocomplete values.** `type="email"`, `type="tel"`, `autocomplete="given-name"`, `postal-code` and `organization` tell an agent exactly what each field wants. - **Only the fields you need.** Every optional field is another chance to guess wrong. Ask for the rest after the lead or booking is in. - **Native controls for choices.** Real `<select>` menus, radio buttons and checkboxes, not custom dropdowns built from `<div>`s. - **Errors written as text next to the field.** "Enter a phone number with area code" tells the agent what to fix. A red border doesn't. - **A clear confirmation.** After submitting, show a page or message that says what happened: "Thanks, Sam. Your table for 4 is booked for Friday 10 October at 7:30 pm. Reference NW-4821." The agent can read that back to the person. A toast that disappears after two seconds can't be. ### Sign-up forms For software companies, sign-up is the conversion. An agent asked to "start a trial" needs to find the sign-up page, fill it in and reach a confirmed account, or at least a clear next step for the person. - **Link to sign-up from every page.** A real link with clear text ("Start free trial"), not a button that opens a modal only after a script loads. AgentScore checks that agents can find and press your sign-up button. - **Keep password rules on the page.** State them before the field ("At least 12 characters"), not only in an error after submit. - **Make "Sign up with Google" an option, not the only route.** Third-party sign-in opens a pop-up on another domain that many agents can't or shouldn't use. An email option keeps the door open. - **Don't hide sign-up behind a sales call.** If your plans need a conversation, say so in text, and offer a booking form the agent can complete. ### Email verification and one-time codes An agent working on someone's behalf usually can't read their inbox or their text messages, and you wouldn't want it to have that access by default. So any step that says "enter the code we just sent" is a handoff point. Handle it well: 1. **Let people do as much as possible before verification.** Create the account, save the booking request or record the lead first, then ask for confirmation. If the agent stops at the verification step, the work isn't lost. 2. **Say clearly what happens next.** "We've sent a link to sam@example.com. Click it to activate your account." The agent can pass that message straight to the person. 3. **Prefer a link over a short-lived code.** A link the person clicks later works across devices and doesn't expire while the agent reports back. If you use codes, give them a sensible lifetime and a "resend" button with a name. 4. **Mark code fields properly.** `autocomplete="one-time-code"` and `inputmode="numeric"` on a single field are easier for people and agents than six separate boxes. ### CAPTCHAs on forms A CAPTCHA on every submission stops AI agents as surely as it stops spam bots. Most sites don't need one on every form. Better options: - **Risk-based challenges** that only appear for suspicious submissions, such as many attempts from one address in a short time. - **Invisible checks** like honeypot fields and rate limits, which stop crude bots without asking anyone to solve a puzzle. - **Recognizing verified agents.** Some agents sign their requests with [Web Bot Auth](https://ghostagentlab.com/blog/signed-requests/), and services such as Cloudflare can let verified bots through. AgentScore checks for a CAPTCHA or challenge when an agent arrives on your site, and at the cart and checkout. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/) for the full picture. ### Booking forms and date pickers Booking is where custom widgets cause the most trouble. A calendar that only works by clicking tiny cells, or a time-slot grid of unlabelled boxes, leaves an agent guessing. | Element | Hard for agents | Easy for agents | | --- | --- | --- | | Date | Custom calendar pop-up with no labels on days | `<input type="date">`, or a calendar whose days are buttons named "Friday 10 October, available" | | Time | Grid of colored boxes | Radio buttons or a list of buttons named "7:30 pm" | | Availability | Grayed-out slots with no text | Disabled options with a reason: "7:00 pm, fully booked" | | Party size or service | Plus and minus icons only | A `<select>` or number field with a label | Also make availability readable as text where you can, for example a list of open times on the page. An assistant answering "is there anything Friday evening?" can then answer without driving the widget at all. ``` <fieldset> <legend>Time on Friday 10 October</legend> <label><input type="radio" name="time" value="19:00" disabled> 7:00 pm (fully booked)</label> <label><input type="radio" name="time" value="19:30"> 7:30 pm</label> <label><input type="radio" name="time" value="20:00"> 8:00 pm</label> </fieldset> ``` If your booking runs inside a third-party widget in an iframe, test it. Some booking providers build accessible widgets and some don't, and you inherit whatever they ship. ### Quote and lead forms - **Split long forms into clear steps** with a heading per step and a named "Next" button. Show where the agent is: "Step 2 of 3: Your project". - **Avoid free-text-only questions** where a set of options would do. "Roof type" as radio buttons is easier to answer correctly than a blank box. - **Make consent explicit.** Marketing opt-ins should be unticked checkboxes with clear labels. An agent should only tick them if the person asked it to. - **Give a real response time.** "We'll reply within one working day" is something the agent can tell the person. "Someone will be in touch" isn't much help. ### Plan the handoff to the person Some steps belong to the person: paying a deposit, accepting terms that bind them, confirming an email address, or signing in with a password only they know. Agents generally stop and hand back at these points, and that's how it should be. Your job is to make the handoff smooth: - Keep what the agent entered. A booking or form that survives a pause, ideally at its own URL, lets the person pick up where the agent left off. - Put the handoff step on its own page with a clear heading, such as "Confirm your booking", so the agent knows it has done its part. - Show a summary of everything entered before the final step, so the person can check it quickly. > **A quick test:** ask an AI assistant with browsing to complete your main form using test details, and watch where it hesitates. Then try the same journey with only the keyboard. The two usually get stuck in the same places. For a structured approach, see [how to test your key journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). ### How Ghost Agent Labs helps [AgentScore](https://ghostagentlab.com/agentscore/) checks the parts of this you can see from the outside: form fields are labelled, buttons and links have names, pop-ups can be dismissed, there's no CAPTCHA on arrival, and agents can find your sign-up or add-to-cart button. [Ghost Agents](https://ghostagentlab.com/ghost-agent/) go further: they run the journey itself, such as sign-up or a contact form, with test data you supply, on a schedule, and show a step-by-step replay when it breaks. They don't submit payments, and the sign-up template stops short of email confirmation. Forms are only half of task completion. For stores, the other half is [checkout](https://ghostagentlab.com/articles/agent-ready-checkout/) and [guest checkout](https://ghostagentlab.com/articles/guest-checkout-ai-agents/). ### Sources and further reading - [HTML Standard: Form control infrastructure (autofill)](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html) (WHATWG) - [HTTP Message Signatures for automated traffic Architecture (Web Bot Auth draft)](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/) (IETF) --- ## How to test your key journeys with AI agents > Which journeys to test with AI agents, how to test by hand and with Ghost Agents, what to measure, and how to test safely without real purchases. Published October 9, 2026 by Ghost Agent Labs · Task completion · https://ghostagentlab.com/articles/test-journeys-with-ai-agents/ ### Key takeaways - Passing technical checks does not prove agents can finish a task, so the only way to know is to send one through. - Treat a flaky result as a real problem, because a journey that passes two times in three fails some customers' assistants. - Pick your most valuable journey, try it three times with an AI assistant, and never let tests make real purchases. Technical checks tell you whether AI agents should be able to use your site. The only way to know whether they can is to send one through and watch. This guide covers which journeys to test, how to test them by hand with the assistants your customers already use, how to automate it, and how to keep testing safe. ### Why checks aren't enough A site can pass every check for labels, named buttons and structured data and still lose agents at a custom size picker, a cookie banner that reappears on the cart page, or a checkout step that needs an account. Those failures only show up when an agent actually tries to finish the job. That's why task completion carries the most weight in [agent readiness](https://ghostagentlab.com/articles/agent-readiness/), and why it can only be measured by running real agents. ### Step 1: Pick the journeys that make money Don't test everything. Start with the two or three journeys where an agent failure costs you a customer: | Type of site | Journeys to test first | | --- | --- | | Online store | Search for a product, choose options and add to cart, check out as a guest up to payment | | Software company | Find pricing, start a trial or sign up, book a demo | | Service business | Find prices and availability, book an appointment, request a quote | | Any site | Find a policy (returns, shipping, cancellation), use the contact form | Write each one the way a customer would ask an assistant: "Find a waterproof jacket in medium under $150 and add it to the cart." Specific goals give clearer results than "buy something". ### Step 2: Test by hand with consumer assistants The quickest start costs nothing. Several consumer AI assistants can now browse and act on websites, usually in an "agent" or browsing mode. Use the ones your customers are likely to use. 1. **Give it the goal and your site.** "On northwind.example, find a 12-cup coffee maker and add it to the cart. Don't buy anything." 2. **Watch, don't help.** Most agent modes show what the agent is doing. Note every hesitation, wrong click and retry, not just whether it finished. 3. **Ask it what went wrong.** If it fails, ask what it was trying to do and what it couldn't find. The answer often names the exact control. 4. **Repeat it.** Agents don't behave the same way every time. Run each journey at least three times before drawing conclusions. Manual testing is good for discovery: it shows you how agents experience your site and builds a shared picture for the team. It doesn't scale, and it won't tell you when something breaks next Tuesday. ### Step 3: Automate it with Ghost Agents [Ghost Agents](https://ghostagentlab.com/ghost-agent/) are AI agents that run your key journeys for you. In the Ghost Agent Labs app, you set up a test for a site you've added: - **Goal.** Start from a common journey (checkout, product search, sign-up or contact form) or describe your own in plain words. - **Start page.** Where the agent begins. It can only visit pages on your site. - **When it passes.** A check on the final page the agent reaches: it reached the payment step, the page shows certain text, or the address contains a certain path. These are checked directly, not by asking an AI model whether it succeeded. - **Test data.** A test email address and any other values the journey needs. - **Schedule and repeats.** Run it by hand, daily or hourly, with several runs per check. Each run records every step: the page the agent saw, what it did and its reasoning. When a run fails, the replay opens on the step where it got stuck, and you can compare it with the last run that passed. Failing tests can alert your team by email or Slack. ### What to measure - **Pass rate.** Because agents vary, one run proves little. Run each check several times and look at the share that passed. Ghost Agents report a journey as passing, flaky (some runs passed) or failing (none did). - **Where it fails.** The step where failures cluster is your fix list: the size picker, the cookie banner, the account wall. - **Steps taken.** A journey that passes in 12 steps one week and 25 the next has become harder, even if it still passes. - **Change over time.** A journey that passed yesterday and fails today usually means something on your site changed: a theme update, a new app, a new pop-up. > **Flaky is a finding, not noise.** If an agent completes a journey two times in three, a real customer's assistant fails one time in three. Treat a flaky result as a problem to fix, and look at the failed runs to see what was different. ### Keep testing safe Testing with agents means letting software click buttons on your live site. Set some rules: - **No real purchases.** Stop at the payment step. Ghost Agents stop when they reach a payment form, never enter card details, and refuse buttons that place orders, pay, or delete or cancel accounts. When testing by hand, tell the assistant not to buy, and stay in control at payment. - **Test data only.** Use a dedicated test email address, such as one on a domain you control, and dummy names and phone numbers. Never use real card or bank details, or a real customer's details. - **Watch your side effects.** Test sign-ups create accounts, test leads land in your CRM and test bookings take slots. Use recognizable test values (a "test+" email or the name "Ghost Test") so your team can filter them out, and avoid submitting booking forms that hold real inventory. - **Tell the people who need to know.** Let whoever runs bot protection and analytics know tests are coming. Ghost Agents identify themselves with `GhostAgent/1.0` in the user agent and sign their requests with Web Bot Auth, so they can be recognized and filtered from reports. - **Consider staging for risky journeys.** If a journey can't be tested on the live site without real consequences, test it on a staging copy that matches production. ### Turn results into fixes A failed run is only useful if someone acts on it. For each failure, record the journey, the step, what the agent was trying to do and a screenshot, then route it to whoever owns that part of the site: the developer for an unlabelled button, the platform admin for forced account creation, marketing for the pop-up. The replay usually tells you which. Most fixes are the same ones covered elsewhere in these guides: [buttons and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/), [cookie banners and pop-ups](https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/), [checkout](https://ghostagentlab.com/articles/agent-ready-checkout/) and [sign-up and booking forms](https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/). ### Where to start this week 1. Run [AgentScore](https://ghostagentlab.com/agentscore/) on your home page to clear the obvious blockers first. 2. Pick your single most valuable journey and try it by hand with an AI assistant, three times. 3. Set it up as a Ghost Agent test with a daily schedule and a pass condition on the final page, so you hear about it the day it breaks. ### Sources and further reading - [HTTP Message Signatures for automated traffic Architecture (Web Bot Auth draft)](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/) (IETF) --- ## Carts AI agents can manage: quantities, promo codes and totals > How to build a cart AI agents can change and check: named quantity and remove controls, usable promo code fields, clear totals and carts that last. Published October 9, 2026 by Ghost Agent Labs · Task completion · https://ghostagentlab.com/articles/cart-ai-agents/ ### Key takeaways - The cart is where an agent checks its work, so items, quantities, discounts and totals must all be readable as text. - Unnamed plus, minus and remove icons, vague promo code errors and carts that empty quickly can stall an order you already won. - Try a five-minute test: change a quantity, remove an item and apply a code using only the keyboard, then read the cart aloud. The cart is where an AI agent checks its work. A person asks for "two bags of the decaf, and use my code SPRING10", and the agent has to change a quantity, apply a code and read back a total the person will agree to pay. If the cart makes any of that hard, the agent guesses or gives up. This guide covers what a cart needs so agents can manage it reliably. ### Why the cart matters more for agents A person glances at a cart and knows whether it looks right. An agent has to read it. Before it hands over for payment, a careful agent reports back: what's in the cart, how many, what it costs and when it arrives. It gets every one of those facts from your cart page. If the cart can't be read, the agent can't confirm, and the order stalls at the point where you've already done the hard work of winning it. For the whole journey from search to payment, see [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/). This guide zooms in on the cart itself. ### Quantity controls The most common problem is a pair of "−" and "+" icons with no names, next to a number that's styled text rather than a field. A person sees a stepper. An agent sees two unnamed buttons and a number it can't change directly. - **Use a labelled number field.** A real `<input type="number">` whose label names the item lets an agent type the quantity it wants. - **Name the stepper buttons.** If you keep plus and minus buttons, give each a name: "Increase quantity of Ethiopia Yirgacheffe". - **Say limits in words.** "Limit 4 per order", next to the field. An agent that asks for 6 and silently gets 4 will tell the person the wrong thing. - **Confirm every change.** Update the line total and the cart total on the page, and announce it in a status message, so the agent knows the change took. - **Make "Update cart" a real button** if your cart needs one, placed next to the quantities, not hidden at the bottom. A cart line that does all of this: ``` <li> <a href="/products/ethiopia-yirgacheffe?variant=340g">Ethiopia Yirgacheffe, 340 g</a> <label for="qty-1">Quantity</label> <input id="qty-1" name="qty-1" type="number" min="1" max="4" value="2"> <span>Limit 4 per order</span> <span>$36.00</span> <button type="button" aria-label="Remove Ethiopia Yirgacheffe, 340 g"> <svg aria-hidden="true">…</svg> </button> </li> <p role="status">Quantity updated to 2. Subtotal $36.00.</p> ``` ### Remove buttons - **Name what each one removes.** On a cart with three items, three buttons called "×", or nothing at all, leave the agent guessing which is which. "Remove Ethiopia Yirgacheffe, 340 g" leaves no doubt. - **Confirm in text.** "Ethiopia Yirgacheffe removed. Undo." A line that simply vanishes could mean it was removed, or that the page is still loading. - **Don't make zero the only way out.** Setting the quantity to 0 should work, but a clear remove button shouldn't be missing. AgentScore's "Buttons and links have names" check looks for exactly this kind of icon-only control on your home page. Apply the same rule in the cart, where it matters more. ### Promo code fields People often ask an assistant to use a code they already have. That only works if the agent can find the field, apply the code and tell whether it worked. - **Give the field a visible label**, such as "Discount code", not just placeholder text. Placeholders vanish as soon as typing starts, and many agents ignore them. - **Pair it with a real "Apply" button.** Not a form that only submits when someone presses Enter. - **If it's collapsed behind "Have a code?"**, make that a real button that says it opens something (`aria-expanded`), not a styled line of text. - **Explain failures in words.** "SPRING10 expired on March 31" or "This code doesn't apply to sale items" lets the agent tell the person why. "Invalid code" in red does not. See [error messages AI agents can understand](https://ghostagentlab.com/articles/error-messages-ai-agents/). - **Show the applied discount as its own line**, with the code name, the amount and a named button to remove it. ### Totals the agent can repeat back The goal is simple: an agent should be able to say "The total is $44.18, including $5.00 shipping and $3.18 tax" and be right. That means each part of the total is on the page as text. | Line | What to show | | --- | --- | | Subtotal | The sum of the items, as text, updated after every change | | Discounts | Each code or promotion by name, with its amount | | Shipping | The cost for the chosen method, or an estimate with the method named. If it truly depends on the address, say "Calculated at checkout" | | Tax | The amount, or "Calculated at checkout". Say whether prices already include tax | | Total | The final figure with its currency | - Show the currency clearly. "$" alone is ambiguous on a site that sells in several countries. See [international sites for AI agents](https://ghostagentlab.com/articles/international-sites-ai-agents/). - Put free-shipping thresholds in words: "Spend $12.00 more for free shipping." A progress bar alone tells an agent nothing. - Keep prices as text in the page, not in images or drawn on a canvas. See [prices AI agents can read](https://ghostagentlab.com/articles/machine-readable-prices/). ### Carts that survive the handoff Most agents hand back to the person to sign in or pay, and that can happen hours later, or on another device. A cart that empties in the meantime undoes the agent's work. - **Keep carts for days, not minutes.** Don't clear them on a short session timeout. - **Keep the cart at a stable URL**, such as `/cart`, as a full page as well as any slide-out drawer. - **Consider cart links.** Some platforms can rebuild a cart from a URL (Shopify calls these cart permalinks), which lets an agent hand the person a link that opens the same cart. Ask your platform what it supports. - **Explain changes.** If stock or a price changes while the cart waits, say so on the page: "Only 1 left. We've changed your quantity to 1." ### Upsell pop-ups and cart drawers An upsell window after "Add to cart", or on the way to checkout, is one more thing an agent has to get past. - If you use one, make it a proper dialog with a heading, a "No thanks" button with text, and a named close button. See [cookie banners and pop-ups](https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/). - **Never pre-add extras**, such as a pre-ticked protection plan. An agent may not notice, and the person pays for something they didn't ask for. In some markets, such as the EU, consumer law already restricts pre-ticked extras. - Recommendations under the cart are fine. Keep the checkout button above them. - Give a cart drawer a labelled close button and a real link to the full cart page. ### What AgentScore checks on your cart The free [AgentScore](https://ghostagentlab.com/agentscore/) scan is read-only: it loads pages but never adds anything to a cart. On many stores the cart is empty when it looks, and the scan says so rather than guessing. For stores, these checks cover the cart: - **AI agents can open your product, pricing and cart pages.** It requests your cart as ChatGPT-User, Claude-User and Perplexity-User and compares what they get with what a browser gets. - **AI agents aren't blocked or shown a CAPTCHA at the cart and checkout.** - **Cart and checkout fields are labelled for agents.** Labels rather than placeholders, the right input types, and autocomplete. If the empty cart shows no fields, this is marked "not tested". - **Shoppers can check out without an account.** It reads the cart for wording like "Sign in to check out" or "Check out as a guest". - **Key pages are linked from the home page**, including the cart, with ordinary links in the HTML. Changing quantities, applying codes and reading totals need items in the cart, so they belong to Task completion, which the free scan marks "not tested". To test them, set up a [Ghost Agent](https://ghostagentlab.com/ghost-agent/) test with a goal in your own words, such as "Add two bags of decaf, apply the code TEST10 and go to checkout", using a test code you've created, and a pass condition that looks for the code's name on the final page. Ghost Agents stop at the payment form, so no order is placed. See [how to test your key journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). > **A five-minute cart test.** Add two items, then use only the keyboard to change a quantity, remove an item and apply a code. Then read the cart aloud as if over the phone: items, quantities, discount, shipping, tax, total. Anything you had to look at a picture to work out, an agent can't see. ### Sources and further reading - [Accessible Rich Internet Applications (WAI-ARIA) 1.2](https://www.w3.org/TR/wai-aria-1.2/) (W3C) - [Create cart permalinks](https://shopify.dev/docs/apps/build/checkout/create-cart-permalinks) (Shopify Dev Docs) - [Directive 2011/83/EU on consumer rights](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32011L0083) (EUR-Lex (European Union)) --- ## Order tracking, returns and reorders by AI agents > How to let AI agents track orders, start returns and reorder for customers without a login, and when to hand back to the person for security. Published October 9, 2026 by Ghost Agent Labs · Task completion · https://ghostagentlab.com/articles/order-tasks-ai-agents/ ### Key takeaways - Letting agents track orders, start returns and reorder without a login can answer common customer questions before they become support tickets. - Match security to the risk, so an order number and email unlock tracking while only the person changes addresses or saved cards. - Start with a public order lookup page and a direct status link in every order email, which needs no new security thinking. Buying is only part of what people ask AI assistants to do. "Where's my order?" "Return the shoes, they're too small." "Order the same coffee again." These after-sale tasks are frequent, they cost you nothing when customers can do them on their own, and they're often locked behind a login an agent can't use. This guide covers how to make them work for AI agents, and where the agent should hand back to the person. ### Why after-sale tasks matter "Where is my order?" is one of the most common reasons customers contact an online store. When an assistant can answer it from your site, that's a support ticket that never gets written. When it can't, the person contacts you anyway, or the assistant pieces together an answer from a carrier's site and gets it wrong. Returns matter before the sale, too. Assistants read return policies when comparing stores, and a clear, easy returns process is part of what makes a store safe to recommend. See [policy pages AI assistants can quote](https://ghostagentlab.com/articles/policy-pages-ai-assistants/). ### Order tracking without a login An agent can't sign in as the person, but it can usually be given an order number and an email address. That's enough for a well-built order lookup. - **Give order lookup its own page** at a stable URL, such as `/orders/lookup`, linked from your footer and help pages with an ordinary link. - **Ask for two things:** the order number and the email address used at checkout. Label both fields, and say where to find the number: "It starts with NW- and is in your confirmation email." - **Show the status in words.** "Shipped on October 3 with UPS. Expected Tuesday, October 7", the tracking number as a link, and the items in the order. A progress bar alone tells an agent nothing. - **Put a direct link in every order email** that opens the order's status page without signing in. Many platforms offer one. Agents reading an email for the person can follow it straight to the answer. - **Keep failures helpful but safe.** "We couldn't find an order matching those details" is right. Don't say which of the two was wrong, which would let anyone test email addresses. - **Limit guessing with rate limits, not a CAPTCHA on every lookup.** Answer too many attempts with a 429 status. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). A lookup form agents can fill in: ``` <form action="/orders/lookup" method="post"> <label for="order">Order number</label> <input id="order" name="order" autocomplete="off" aria-describedby="order-hint"> <p id="order-hint">Starts with NW-. You'll find it in your confirmation email.</p> <label for="email">Email used at checkout</label> <input id="email" name="email" type="email" autocomplete="email"> <button type="submit">Find my order</button> </form> ``` Show only what's needed on a page reached this way: status, items, carrier and the delivery city. Leave out the full address, phone number and payment details. ### Returns and exchanges - **Start from a plain-text policy**: how long customers have, what condition items must be in, who pays return shipping, how long refunds take and what can't be returned. - **Open the returns portal with an order number and email**, the same as tracking. If returns need an account, most agents stop at the door. - **Label every choice.** Items as checkboxes named with the product and size, the reason as a list of options in words, and refund, exchange or store credit as radio buttons that say what each one means: "Store credit: issued as soon as we receive the item." - **Explain why an item isn't eligible.** "This item was delivered 45 days ago. Returns close after 30 days." See [error messages AI agents can understand](https://ghostagentlab.com/articles/error-messages-ai-agents/). - **Summarize before submitting.** A review step ("You're returning 1 item for a $64.00 refund to your original payment method") lets the agent check with the person first. - **Say what happens next in words.** Where the label is (a link to the PDF, and "We've emailed it to you"), drop-off options, and when the refund will arrive. ### Reorders "Order the same again" usually needs order history, and order history usually needs an account. You can still make it easy: - **Add a "Buy again" button to the order status page**, the one reached from the email link. Name it after the item: "Buy again: Ethiopia Yirgacheffe, 340 g". - **Put the same item back in the cart**: the same size, color and quantity. Say if anything changed: "The price is now $19.00" or "This size is out of stock." - **Keep product URLs stable.** A product link in a year-old confirmation email should still work, or redirect to the replacement product with a note saying so. See [canonical URLs and duplicate pages](https://ghostagentlab.com/articles/canonical-urls-ai-agents/). - **Make subscription controls clear** if you sell subscriptions. "Skip next delivery" and "Change quantity" as named buttons, with the next delivery date in words. ### Security, accounts and handing back to the person These tasks touch personal details and money, so there's a real tension. Agents shouldn't need the person's password, and anyone holding an order number shouldn't be able to change an order. The answer is to match the check to the risk: | Task | What it should take | Who finishes it | | --- | --- | --- | | Track an order | Order number and email | The agent | | Check return eligibility and options | Order number and email | The agent | | Submit a return or exchange | Order number and email, plus a review step | The agent, once the person agrees | | Reorder | A "Buy again" link or button | The agent builds the cart, the person pays | | Change a delivery address or cancel | A one-time code or emailed link | The person | | Change email, password or saved cards | Full sign-in, ideally with two-step sign-in | The person only | - **Use step-up checks for risky actions only.** A one-time code sent by email or text is a clean handoff point: the agent stops and says "Northwind has sent a code to your email", and the person takes it from there. - **Make the handoff obvious.** Browser agents typically hand control back for sign-in and payment. A sign-in form with labelled fields, and support for passkeys or password managers, makes that moment quick for the person. - **Never ask for card details to prove who someone is.** No one, human or agent, should be typing card numbers to see an order status. - **Keep help content public.** Delivery times, return policy and FAQs don't need a login. See [login walls and gated content](https://ghostagentlab.com/articles/login-walls-ai-agents/). The same thinking applies to buying: see [guest checkout](https://ghostagentlab.com/articles/guest-checkout-ai-agents/) and [sign-up and booking forms agents can use](https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/). ### How to test these journeys 1. **Try it yourself with an assistant.** Place a test order, then ask an assistant that can browse to find its status using the order number and email. Watch where it stalls. 2. **Check the basics with [AgentScore](https://ghostagentlab.com/agentscore/).** It checks that AI assistants are let in and can open your key product, pricing and cart pages. It doesn't test order lookup or returns, and the free scan marks Task completion "not tested". 3. **Monitor them with [Ghost Agents](https://ghostagentlab.com/ghost-agent/).** Describe the journey in your own words, such as "Find the order tracking page, look up order NW-10422 with test@northwind.example, and report the delivery status", and set a pass condition that looks for text like "Shipped" on the final page. For returns, write the goal so the agent stops at the review step, without submitting. > **Start with tracking.** It's the most common after-sale question, it needs no new security thinking, and a public lookup page with a direct link in every order email works for customers and their agents alike. ### Sources and further reading - [RFC 6585: Additional HTTP Status Codes](https://www.rfc-editor.org/rfc/rfc6585.html) (IETF) --- ## Why your analytics can't see AI agents, and how to measure agent traffic > JavaScript analytics tags miss most AI agents. How to measure agent traffic from server and CDN logs, and what to track: who's visiting, who's real, and where they're blocked. Published October 8, 2026 by Ghost Agent Labs · Measurement · https://ghostagentlab.com/articles/measure-ai-agent-traffic/ ### Key takeaways - Tools like Google Analytics mostly cannot see AI agents, because most agents never run the tag and known bots are filtered out. - Counting visits by agent name alone lets impostors inflate your numbers, so verify agents against operators' published IP ranges or reverse DNS. - Get access to your server or CDN logs, then track which agents visit, what they read and where they are blocked. Ask most businesses how many AI agents visited their site last month and they can't say. It's not that the agents aren't coming. It's that the tools most teams use to count visitors were built in a way that can't see them. Here's why, and how to measure agent traffic properly. ### Why analytics misses agents Tools like Google Analytics count a visit when a small JavaScript tag runs in the visitor's browser and reports back. That works for people. It fails for AI agents for three reasons: - **Most agents never run the tag.** AI crawlers and assistant fetchers download the HTML and leave without running JavaScript, so the tag never fires. - **Known bots are filtered out on purpose.** Analytics tools exclude known bot traffic so it doesn't distort your human numbers. That's right for marketing reports, but it hides exactly the visits you want to see. - **Browser agents blend in.** Agents that do run JavaScript often look like ordinary browsers, so they're counted as people, mixed in with everyone else. You may see AI in your analytics as referral traffic: people who click a link in a ChatGPT or Perplexity answer and arrive on your site. That's valuable, but it's the result of agent visits, not the visits themselves. It doesn't show which agents read your pages, which they couldn't, or what they saw. ### Where agent visits do show up Every request to your site, from a person or an agent, passes through your web server or CDN, and they can log it whether or not any JavaScript runs. Each log line typically includes: | Field | What it tells you | | --- | --- | | User agent | Which agent it claims to be | | IP address | Where it came from, which you need to verify it's real | | Path | Which page or file it asked for | | Status code | Whether it got the page (200), was blocked (403), rate limited (429) or hit a missing page (404) | | Time and bytes | When it came, and how much content it received | ### Getting your logs - **Cloudflare:** Logpush sends HTTP request logs to storage or another service. Availability depends on your plan. - **Vercel:** log drains stream request logs to an endpoint you choose. - **Fastly, Akamai and Amazon CloudFront** all offer access logs or real-time log streaming. - **Your own servers:** Nginx and Apache write access logs by default. - **Hosted platforms** such as Shopify don't usually give you raw request logs, which makes agent traffic much harder to see. Ask your platform what bot and crawler reporting it offers. ### A quick look with the command line If you have an Nginx or Apache access log in the standard "combined" format, this counts requests from some well-known AI agents: ``` awk -F'"' '{print $6}' access.log \ | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|Amazonbot|meta-external[a-z]+|Bytespider|CCBot' \ | sort | uniq -c | sort -rn ``` It's a useful first look, but it has real limits: it trusts the user agent, so impostors count as the real thing; it only knows the names you list; and it says nothing about whether the agents succeeded. ### What to measure 1. **Who's visiting.** Requests by agent and by kind: training crawlers, AI search, assistant fetchers and browser agents. The mix matters more than the total. Assistant fetchers are people asking about you right now. 2. **Who's real.** Verify each agent against its operator's published IP ranges or reverse DNS, and separate verified, spoofed and "can't tell". See [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). Without this, scrapers inflate the numbers. 3. **What they're reading.** The pages agents request most. Are they reaching products, pricing and policies, or stuck on the home page and old URLs? 4. **Whether they're blocked.** The share of agent requests that get 403, 429 or challenge responses. A rise usually means a bot protection change caught good agents. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/). 5. **Errors.** 404s for agents often point to old URLs that AI models learned and still request. Redirect them. 6. **Trends.** Week over week, by agent. New agents appear often, and a sudden drop from one usually means something on your side changed. ### Privacy Logs contain IP addresses, which are personal data for human visitors. You only need the full detail for bots and agents. Hash or drop IP addresses for human traffic, keep agent records separately, and set a retention period that matches your privacy policy. > **Ghost Agent Labs does this for you.** Connect Cloudflare Logpush, a Vercel log drain, or any JSON log feed. Every request is matched against a registry of more than 1,500 known agents, crawlers are verified against published IP ranges and reverse DNS, and you get a dashboard of who's visiting, which agents are real, what they read, and where they're blocked. Human visits are only counted, never stored. [Start free](https://app.ghostagentlab.com/signup). Measuring traffic tells you who's coming. To find out whether they succeed, test the journeys that matter. See [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/). ### Sources and further reading - [Known bot-traffic exclusion](https://support.google.com/analytics/answer/9888366) (Google Analytics Help) - [Logpush](https://developers.cloudflare.com/logs/logpush/) (Cloudflare Docs) - [Working with Drains](https://vercel.com/docs/drains) (Vercel Docs) - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI) --- ## Tracking visits and sales that come from AI assistants > How to spot visits from ChatGPT, Perplexity and other AI assistants in your analytics, report them in GA4, and understand what attribution misses. Published October 9, 2026 by Ghost Agent Labs · Measurement · https://ghostagentlab.com/articles/ai-referral-traffic/ ### Key takeaways - People who click links in AI assistant answers can be measured in your existing analytics and are often worth reporting separately. - Treat AI referral numbers as a floor, because app visits and copied links arrive as direct traffic and many answers drive no click. - In GA4, create a custom channel group for AI assistants placed above Referral, and check your referral report monthly for new sources. When someone asks ChatGPT, Perplexity or Gemini for a recommendation and clicks a link in the answer, they land on your site as a visitor like any other. Those visits can be measured in the analytics you already have, and they're often worth separating out. This guide shows how to spot them, how to report on them, and what the numbers can't tell you. ### Two different things to measure AI shows up in your data in two ways, and it helps to keep them apart: - **AI agents visiting your site.** Crawlers and assistants that read your pages, mostly without running your analytics tag. These only show up in server or CDN logs. See [how to measure AI agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). - **People arriving from AI assistants.** A person reads an AI answer, clicks a link to your site, and browses in their own browser. Your analytics tag runs normally. This is AI referral traffic, and it's what this guide covers. The first tells you whether AI can read and use your site. The second tells you whether that's turning into visits and sales. ### How AI referrals show up Analytics tools work out where a visit came from in two main ways: the referrer, which the browser sends to say which site the person came from, and campaign tags (UTM parameters) on the link itself. #### Referrers When someone clicks a link in a web-based assistant, the browser often sends that assistant's domain as the referrer. Domains you're likely to see include: | Assistant | Referrer domains you may see | | --- | --- | | ChatGPT | `chatgpt.com` (older visits may show `chat.openai.com`) | | Perplexity | `perplexity.ai`, `www.perplexity.ai` | | Google Gemini | `gemini.google.com` | | Microsoft Copilot | `copilot.microsoft.com` | | Claude | `claude.ai` | Treat this list as a starting point rather than a complete map. Assistants change domains, launch new apps and change how they link out, so check your own referral report for new sources every month or so. #### Campaign tags Some assistants add their own tag to outbound links. ChatGPT, for example, has been adding `utm_source=chatgpt.com` to many links in its answers. In your analytics that shows as the source `chatgpt.com`, sometimes with no medium, or alongside the referrer. Other assistants may do the same, or nothing at all. #### Visits that arrive with no label Not every AI referral announces itself. Mobile and desktop apps often open links without sending a referrer, privacy settings and browsers can strip it, and a person who copies a URL from an answer and pastes it into a new tab arrives as "direct". Some of your direct traffic is AI referral traffic you can't see. Treat any AI referral number as a floor, not a total. ### Reporting on it in Google Analytics 4 Analytics tools have been changing how they group this traffic. In GA4's default channel grouping, AI assistants have typically ended up under Referral, mixed in with every other website that links to you. Check what your property shows today, because default channel definitions get updated. If AI referrals aren't already separated, two approaches work: 1. **A quick view.** In the Traffic acquisition report, change the dimension to "Session source" and filter for the domains above. Good for a one-off look. 2. **A custom channel group.** In Admin, under Data display, create a channel group with an "AI assistants" channel whose rule matches the session source against a pattern, and place it above Referral so it wins. Custom channel groups apply to your reports going back in time, so you see history too. A source pattern to start from: ``` (^|\.)(chatgpt\.com|chat\.openai\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.ai)$ ``` Use a "matches regex" condition on session source. Keep the pattern in one place, such as a shared doc, and add new assistants as they appear in your referral report. Other analytics tools work the same way: a rule on referrer or source that puts these visits in their own channel. > **Google's own AI features are harder.** Clicks from AI Overviews and AI Mode in Google Search generally arrive as ordinary Google organic traffic, and analytics can't separate them from classic search results. Search Console reports them as part of overall Search performance. Don't expect your "AI assistants" channel to include them. ### What to measure - **Sessions and trend.** AI referral sessions by assistant, week over week. The direction matters more than the size today. - **Landing pages.** Which pages assistants send people to. Often it's product pages, comparison content or policy pages rather than the home page. Those pages now act as front doors, so make sure they convert. - **Engagement and conversion.** Conversion rate, revenue and leads from the AI channel compared with search and other referrals. People arriving from an AI answer have often already been given a recommendation, so watch how their behavior compares. - **Assisted conversions.** Someone may first hear about you from an assistant and buy a week later through a branded search. In GA4, compare attribution models or look at conversion paths to see where AI sources appear early in the journey. ### The limits of attribution Be honest with stakeholders about what this data can and can't show: - **It undercounts.** App traffic and copied links arrive as direct. The real number is higher than what you see. - **It only counts clicks.** Many AI answers mention a brand without a click. Someone who reads "Northwind has the best return policy" and later types your address directly is AI-influenced but won't appear here. - **It doesn't count agent purchases.** When an assistant buys through an [agentic commerce protocol](https://ghostagentlab.com/articles/agentic-commerce-protocols/), there may be no web session at all. Those orders show up in your commerce platform, so ask your platform how they're tagged. - **Browser agents blur the line.** An agent browsing on someone's behalf may run your tag and look like a normal visitor. Some of what looks like human traffic may be agents. ### Getting more AI referral traffic Assistants recommend and link to sites they can read and trust. The groundwork is the same as for agent readiness: let AI search and assistant agents in through [robots.txt](https://ghostagentlab.com/articles/robots-txt-ai-agents/), put key content in the HTML, describe products and prices with [structured data](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/), and make policy pages easy to quote. If you block `OAI-SearchBot` or `PerplexityBot`, don't expect referrals from ChatGPT search or Perplexity. Then make sure the visits convert. The person arriving from an AI answer, or the agent acting for them, needs a page that works. [AgentScore](https://ghostagentlab.com/agentscore/) shows how your site looks to AI agents, and [Ghost Agents](https://ghostagentlab.com/ghost-agent/) test the journeys that turn a referral into a sale. ### Sources and further reading - [Custom channel groups](https://support.google.com/analytics/answer/13051316) (Google Analytics Help) - [AI Features and Your Website](https://developers.google.com/search/docs/appearance/ai-features) (Google Search Central) - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI) --- ## Agent readiness KPIs: what to report each month > A monthly agent readiness report in six measures: AgentScore, agent visits, verification, blocks and errors, Ghost Agent pass rates and AI referrals. Published October 9, 2026 by Ghost Agent Labs · Measurement · https://ghostagentlab.com/articles/agent-readiness-kpis/ ### Key takeaways - A one-page monthly report on six measures keeps agent readiness on the agenda, with your own trend as the main comparison. - Single numbers mislead, so read the measures together and never report agent visits from user agent names alone, since impostors use them. - Give each measure one owner and note what changed each month, such as releases, bot protection changes and new apps. Agent readiness work needs a monthly number, or it quietly slides down the priority list. This guide sets out a one-page monthly report built on six measures: where each one comes from, what good looks like, and how to read them together. There are no reliable industry benchmarks for most of these yet, so the comparison that matters is your own trend. ### Four rules for the report - **Trend over level.** A score of 68 means little on its own. A score that went from 52 to 68 in a quarter, with the reasons, means a lot. - **Keep two questions apart.** "Can AI agents use our site?" (readiness, errors, test results) and "Is it turning into business?" (referrals and orders). The first leads; the second follows. - **One owner per measure.** Each number needs someone who explains it and acts on it. See [who owns agent readiness](https://ghostagentlab.com/blog/who-owns-agent-readiness/). - **Note what changed.** Releases, bot protection changes, new apps and campaigns. Most movements in these numbers trace back to one of them. ### The six measures | Measure | Question it answers | Where it comes from | | --- | --- | --- | | AgentScore | Can agents get in, read and find their way around? | An AgentScore scan | | Agent visits by type | Who's visiting, and what are they reading? | Server or CDN logs | | Verification share | How much of that traffic is really who it says it is? | Logs checked against operators' published proof | | Blocks and errors | Are we turning agents away? | Status codes in your logs | | Ghost Agent pass rates | Can an agent actually finish the job? | Ghost Agent tests | | AI referral visits and orders | Is it bringing in customers and revenue? | Your analytics and your store's orders | In Ghost Agent Labs, the site's Overview page brings several of these together. The sections below name the page that holds each one. ### 1. AgentScore **Report:** the overall score and its band, the scores for Access, Readability and Navigability, and how many findings you fixed and how many are new. The free scan doesn't test Task completion yet, so the score covers the other three categories. **Where:** the Readiness page. Every scan is saved, so you can track the score over time, and "All findings" exports to CSV for your developers. **Good looks like:** a steady or rising score, no critical findings open for more than a month, and every drop explained. Scan on the same day each month and after major releases, so the numbers compare like with like. The bands are 80 and up Good, 60 to 79 Fair, 40 to 59 Weak and below 40 Poor. See [reading your AgentScore report](https://ghostagentlab.com/blog/reading-your-agentscore-report/). ### 2. Agent visits by type **Report:** requests by visitor type (search crawlers, AI training crawlers, AI assistants and autonomous agents) compared with last month, the pages agents read most, and any new agents. **Where:** the Agent traffic page ("Requests by visitor type", "Agent sessions" and "Pages agents request most"), and "What changed" on the Overview, which lists first-time agents and agents whose traffic rose or fell by half or more. **Good looks like:** AI assistant and AI search traffic holding steady or growing, and agents reaching product, pricing and policy pages rather than stalling on the home page or old URLs. Don't celebrate total volume. A busy training crawler isn't a customer. Assistant requests are the closest thing to a customer visit from AI, because each one is usually a person's question. See [how to measure AI agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). ### 3. Verification share **Report:** of the requests claiming to be a known agent, the share that were verified, couldn't be verified, and were spoofed. **Where:** the Verification & spoofing page. **Good looks like:** spoofed traffic close to zero, or falling after you block it, and a verified share that rises as more operators publish ways to check their agents. This measure also keeps the rest of the report honest: never report "ChatGPT visited 40,000 times" from user agent names alone, because impostors use those names too. See [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). ### 4. Blocks and errors **Report:** the share of agent requests that were blocked (401, 403 and 429 responses), the 404 and server error rates, and the agents and pages most affected. **Where:** "Errors: agents vs people" on the Overview, "What agents got back" on the Agent traffic page, and the status codes on each agent's page. **Good looks like:** blocked responses to verified AI assistants near zero, agents getting errors about as often as people do rather than much more, and agent 404s trending down as you redirect old URLs. A sudden rise almost always follows a change to bot protection or rate limits. See [finding the errors AI agents hit](https://ghostagentlab.com/articles/agent-errors-in-logs/). ### 5. Ghost Agent pass rates **Report:** for each journey you test, such as checkout, product search or sign-up, whether it's passing, flaky or failing, its pass rate, and the step where failures cluster. **Where:** the Ghost Agents page, with a replay of every run. **Good looks like:** your money journeys passing, flaky results treated as findings rather than noise, and failures fixed within a sprint. A pass rate below 100% means some customers' assistants would fail too. This is the measure closest to revenue, because it tests whether an agent can do the job, not just whether the page looks right. See [how to test your key journeys with AI agents](https://ghostagentlab.com/articles/test-journeys-with-ai-agents/). ### 6. AI referral visits and orders **Report:** visitors who arrived from AI assistants' answers, by assistant, their conversion rate, and orders and revenue from AI. **Where:** your analytics, set up as in [tracking visits and sales from AI assistants](https://ghostagentlab.com/articles/ai-referral-traffic/). In Ghost Agent Labs, "Visitors from AI assistants" is on the Overview. Send your store's orders to the site's orders URL and Mission Control shows "Agent-driven revenue": orders referred by an AI answer or placed by an agent, and their share of revenue. **Good looks like:** growth from whatever base you start at, and conversion from AI referrals that holds up against your other referral sources. Present it as a floor, not a total: app traffic and copied links arrive as "direct", and many AI answers influence a purchase without a click. ### Reading the measures together Single numbers mislead. These combinations are where the report earns its keep: - **Score steady, blocks up.** Something changed in bot protection or rate limits. Check the agent's page for which pages are affected and when it started. - **Visits up, referrals flat.** Assistants are reading you but not sending people. Look at product data, prices and policy pages: are they giving assistants what they need to recommend you? - **Score unchanged, pass rate down.** A journey broke in a way the scan can't see: a new pop-up, a redesigned size picker, a checkout app. The run replay shows where. - **Referrals up, checkout test failing.** AI is sending you customers and losing some of them at the last step. This is the most urgent combination on the page. - **Spoofed share up.** Scrapers are using agent names. It doesn't hurt readiness, but it inflates traffic numbers and costs bandwidth. ### A one-page template 1. **Headline.** One sentence: better, worse or the same, and why. 2. **Scorecard.** The six measures, this month against last month, with an arrow and a short note each. 3. **What changed.** Site releases, bot protection changes and new agents, linked to the numbers they moved. 4. **Fixed this month.** The findings closed and the journeys repaired. 5. **Next month.** The top three fixes, each with an owner. 6. **Decisions needed.** For example, whether to allow a new AI agent, or budget for platform work. > **What to leave out.** Total bot requests without a breakdown, scores without a trend, and industry comparisons nobody can source. They invite the wrong questions. If you're presenting upward, see [how to explain agent readiness to your board](https://ghostagentlab.com/blog/explain-agent-readiness-to-your-board/), and for what to fix first, [the 90-day agent readiness plan](https://ghostagentlab.com/blog/90-day-agent-readiness-plan/). --- ## Finding the errors AI agents hit, in your server and CDN logs > How to find the 403s, 429s, challenges, timeouts and server errors AI agents hit in your logs, with example queries and how to separate real agents from fakes. Published October 9, 2026 by Ghost Agent Labs · Measurement · https://ghostagentlab.com/articles/agent-errors-in-logs/ ### Key takeaways - When AI agents are turned away from your site nobody tells you, but the evidence is in your server and CDN logs. - Some blocked requests are impostors using agent names, so separate verified agents from spoofed ones before loosening any bot protection rule. - Check agent error rates after every bot protection or CDN change and at least monthly, starting with blocks, rate limits and challenges. When an AI agent can't get into your site, nobody tells you. There's no complaint and no support ticket. The assistant just answers with someone else's products. The evidence is in your server and CDN logs. This guide shows which errors to look for, how to find them with a few queries, and how to separate real agents from impostors before you change anything. It builds on [how to measure AI agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/), which covers getting hold of your logs in the first place. ### The errors that matter | What you see | What it usually means for an agent | Who usually fixes it | | --- | --- | --- | | 403 Forbidden | Bot protection, a firewall rule or a country block turned it away | Bot protection or CDN admin | | 401 Unauthorized | The page needs a login | E-commerce or product team | | 429 Too Many Requests | It hit a rate limit | CDN admin or developers | | Challenge page | A "checking your browser" page or CAPTCHA it can't solve. Often a 403 or 503, sometimes a 200 | Bot protection or CDN admin | | 500, 502, 503 | Your server failed. Agents can reach pages people rarely visit, which may not be cached | Developers or hosting | | 504, slow responses, 499 | The page took too long. In nginx, 499 means the client gave up and closed the connection first | Developers or hosting | | 404 Not Found | An old or guessed URL. AI models often remember URLs you've since changed | Content or SEO team | Challenge pages are the easiest to miss, because some bot protection serves them with a 200 status, so they look like successes. Look for your CDN's security or bot action field instead. Cloudflare, for example, adds a `cf-mitigated: challenge` header to challenge responses. A response that's much smaller than the real page usually is one too. ### What your logs need Most logs already have the basics. Check yours include: - **Time, client IP address and user agent.** The IP address is what lets you verify the agent later. - **Path and status code.** Query strings can be dropped; you rarely need them, and they can hold personal data. - **Response size and response time.** These catch challenge pages and timeouts. - **The CDN's security action**, if it has one: allowed, blocked, challenged, rate limited. On nginx, a log format like this captures them: ``` log_format agents '$time_iso8601 $remote_addr "$request_method $uri" ' '$status $body_bytes_sent $request_time "$http_user_agent"'; ``` If you only have the standard "combined" access log, this one-liner counts status codes for three AI assistants: ``` grep -E 'ChatGPT-User|Claude-User|Perplexity-User' access.log \ | awk '{print $9}' | sort | uniq -c | sort -rn ``` ### Example queries Once logs are in a database or log tool, a few queries answer most questions. The examples below are illustrative, written for PostgreSQL against a table called `requests` with columns `ts`, `client_ip`, `user_agent`, `path`, `status`, `bytes`, `duration_ms` and `security_action`. Rename them to match your own log store; the same ideas work in BigQuery, Athena or a log search tool. #### Error rates by agent, last 7 days ``` SELECT CASE WHEN user_agent ILIKE '%ChatGPT-User%' THEN 'ChatGPT-User' WHEN user_agent ILIKE '%OAI-SearchBot%' THEN 'OAI-SearchBot' WHEN user_agent ILIKE '%Claude-User%' THEN 'Claude-User' WHEN user_agent ILIKE '%Perplexity-User%' THEN 'Perplexity-User' WHEN user_agent ILIKE '%PerplexityBot%' THEN 'PerplexityBot' WHEN user_agent ILIKE '%Googlebot%' THEN 'Googlebot' ELSE 'everything else' END AS agent, COUNT(*) AS requests, COUNT(*) FILTER (WHERE status IN (401, 403)) AS blocked, COUNT(*) FILTER (WHERE status = 429) AS rate_limited, COUNT(*) FILTER (WHERE status = 404) AS not_found, COUNT(*) FILTER (WHERE status >= 500) AS server_errors, ROUND(100.0 * COUNT(*) FILTER (WHERE status >= 400) / COUNT(*), 1) AS error_pct FROM requests WHERE ts >= now() - interval '7 days' GROUP BY 1 ORDER BY requests DESC; ``` "Everything else" mixes people with other bots, many of them scanners probing for pages that don't exist. For a fair baseline, compare with requests from ordinary browsers: if AI agents get errors far more often than people do, something is turning them away. #### Where AI assistants are turned away ``` SELECT path, status, COUNT(*) AS hits FROM requests WHERE ts >= now() - interval '7 days' AND user_agent ~* '(ChatGPT-User|Claude-User|Perplexity-User)' AND (status IN (401, 403, 429) OR security_action = 'challenge') GROUP BY path, status ORDER BY hits DESC LIMIT 20; ``` #### Challenges and timeouts by day ``` SELECT date_trunc('day', ts) AS day, COUNT(*) FILTER (WHERE security_action = 'challenge') AS challenged, COUNT(*) FILTER (WHERE status IN (499, 504)) AS timed_out, COUNT(*) FILTER (WHERE duration_ms > 10000) AS over_10s FROM requests WHERE user_agent ~* '(ChatGPT-User|Claude-User|Perplexity-User|OAI-SearchBot)' GROUP BY 1 ORDER BY 1; ``` A step change on one day usually lines up with a release or a bot protection change. That's your first lead. ### Separating verified agents from claimed ones Everything above trusts the user agent, and a user agent is only a claim. Scrapers often call themselves ChatGPT-User or Googlebot to get past bot protection. When your firewall blocks them, that 403 is working as intended. So before you loosen any rule, split agent errors into three groups: - **Verified.** The request came from the operator's published IP ranges, passed forward-confirmed reverse DNS, or carried a valid signature (Web Bot Auth, still an emerging standard). Errors here are real problems to fix. - **Spoofed.** It used a known agent's name but failed the operator's check. Blocking these is correct. - **Unverifiable.** The operator doesn't publish a way to check. Decide case by case. Operators such as OpenAI publish their agents' IP ranges (see [OpenAI's crawler documentation](https://platform.openai.com/docs/bots)). Load them into a table, refresh it regularly, and join: ``` -- agent_ranges(agent text, cidr cidr), loaded from each operator's published list SELECT r.agent IS NOT NULL AS verified, l.status, COUNT(*) AS hits FROM requests l LEFT JOIN agent_ranges r ON l.client_ip::inet << r.cidr AND l.user_agent ILIKE '%' || r.agent || '%' WHERE l.user_agent ~* '(ChatGPT-User|OAI-SearchBot)' AND l.ts >= now() - interval '7 days' GROUP BY 1, 2 ORDER BY 1 DESC, hits DESC; ``` For the methods in detail, including reverse DNS and signed requests, see [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). > **Mind the privacy.** Logs hold IP addresses, which count as personal data for human visitors. Filter to agent user agents before you keep detailed records, hash or drop IP addresses for everyone else, and keep a retention period that matches your privacy policy. ### From errors to fixes - **Verified assistants get 403s on product or policy pages:** allow verified bots in your bot protection. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/) and [setting up your CDN for AI agents](https://ghostagentlab.com/articles/cdn-settings-ai-agents/). - **Repeated 429s:** give verified agents their own limits, and send `Retry-After`. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). - **Blocks only from certain countries or networks:** see [geo-blocking, VPN blocks and AI agents](https://ghostagentlab.com/articles/geo-blocking-ai-agents/). - **Challenges at the cart or checkout:** these cost orders directly. AgentScore checks for them, and [the checkout guide](https://ghostagentlab.com/articles/agent-ready-checkout/) covers the fix. - **The same 404s week after week:** redirect old URLs to their current pages. - **Server errors and timeouts:** cache the pages agents request most, and check them in your monitoring like any other. If writing queries isn't your team's idea of a good week, Ghost Agent Labs does this from your Cloudflare, Vercel or JSON logs. The Agent traffic page shows what agents got back, each agent's page shows its status codes, the Verification & spoofing page splits verified from spoofed, and Alerts tell you when a major AI assistant starts being blocked. Either way, check after every bot protection or CDN change, and at least monthly as part of your [agent readiness KPIs](https://ghostagentlab.com/articles/agent-readiness-kpis/). ### Sources and further reading - [Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) (OpenAI) - [Detect a Challenge Page response](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/) (Cloudflare Docs) - [Module ngx\_http\_log\_module](https://nginx.org/en/docs/http/ngx_http_log_module.html) (nginx) - [RFC 6585: Additional HTTP Status Codes](https://www.rfc-editor.org/rfc/rfc6585.html) (IETF) --- ## What scanning 600 stores taught us about our own scanner > Running AgentScore on 600 real stores exposed three bugs in our own scanner, and one run that looked perfect because almost nothing was tested. What we found, fixed and changed. Published October 9, 2026 by Ghost Agent Labs · Engineering · https://ghostagentlab.com/blog/what-scanning-600-stores-taught-us/ To write [our study of online stores](https://ghostagentlab.com/blog/agentscore-store-study-2026/), we ran AgentScore on 600 real websites in one day. The study taught us a lot about stores. It taught us even more about our own scanner: three bugs that tests on our own fixture sites never caught, and one run whose results looked perfect because almost nothing had been tested. Here's what we found, what we changed, and what it means for your score. ### Why run the scanner at scale Every AgentScore check is tested against small, carefully built example sites. That proves the check does what we meant. It doesn't prove we meant the right thing for the thousands of ways real stores are built. Scanning 600 stores, from many countries, platforms and sizes, was the first time AgentScore met that variety all at once, and the first time we read its results in aggregate, where a check that's wrong for many stores stands out. ### Bug 1: AgentScore only spoke English Several checks work by reading what a button or link says: finding the add-to-cart button, recognising the button that closes a cookie banner, spotting the link to the cart. All of them only knew English words. On a German store, "In den Warenkorb" wasn't recognised as an add-to-cart button, and "Alle akzeptieren" didn't count as a way to close the banner. In our first run, 61 stores whose pages weren't in English were reported as having no add-to-cart button at all. When we rescanned eight of them with the fix, AgentScore found the button on six. The other two turned out to be a different problem (see bug 3). There was a subtler issue underneath. The usual way to match whole words in code only understands the 26 letters of the English alphabet, so even a correctly translated phrase could fail to match at the edges of words in Russian or with accented letters. Japanese, Chinese and Korean don't put spaces between words at all. **What we changed:** the words AgentScore looks for now live in one list covering 19 languages, including German, French, Spanish, Italian, Dutch, the Nordic languages, Polish, Turkish, Russian, Ukrainian, Japanese, Chinese and Korean, and matching works in every alphabet. It also changed which sites AgentScore recognises as stores, because finding the cart link is part of that: the same sample produced 230 stores instead of 219. ### Bug 2: Real buttons reported as fake ones AgentScore checks that your add-to-cart button is a real button, because agents look for buttons and links, not for text that happens to be clickable. To find the button, it looked for the smallest element whose text matched. Many shop themes write their buttons like this: ``` <button type="submit"> <span>Add to cart</span> </button> ``` The smallest element containing "Add to cart" is the `<span>`, so AgentScore reported "a plain \<span>, not a real button", on a perfectly good button. That's a false alarm, and an expensive one: in the study's second run, 57% of the stores where we found the button were flagged. After the fix, the true figure is 24%. **What we changed:** when the matching text sits inside a real button or link, AgentScore now judges that button or link. A `<div>` styled to look like a button is still caught, because that really is a problem for agents. ### Bug 3: Sold-out products counted against stores AgentScore checks one product page per store. When that product was sold out, its buy button had been replaced by a disabled "Sold out" button, "売り切れ" on a Japanese store or "Agotado" on a Spanish one, and AgentScore reported that it couldn't find an add-to-cart button. That says nothing about the store's buttons, only about its stock. **What we changed:** AgentScore now recognises a sold-out product, from its button in any of the 19 languages or from the product data saying it's out of stock. Then it does what a shopper would: it tries up to two other products. If they're all sold out, the check is marked as not tested, which never costs points. In the study, AgentScore moved past a sold-out product on 4 stores and skipped the check on 5 where everything it tried was sold out. ### The run that looked perfect One of our runs came back looking wonderful. Navigation scored 100%. Not one store had an unnamed button. No pop-ups blocked anything. It was wrong. After an update to how AgentScore opens pages in its browser, the browser on our scanning machine couldn't complete secure connections, and it couldn't load a single page. Every check that needs a real browser was skipped, on 245 of 245 stores. Skipped checks don't cost points, by design, so the scores went up instead of down. We caught it because the numbers were too good to be true, not because anything failed. That's not good enough, so we added a safeguard: our analysis now refuses to produce results if more than 10% of stores couldn't be opened in a browser. We fixed the scanning machine's setup, confirmed one scan by hand, and ran all 600 sites again. > **Your own scans are protected the same way.** When AgentScore can't open your pages in a browser, your report says so and marks those checks as not tested. It never presents a skipped check as a pass. See [how to read your AgentScore report](https://ghostagentlab.com/blog/reading-your-agentscore-report/). ### What this means for your score - **Stores outside English-speaking markets** get a fairer score, and some will see it rise, as buttons, banners and cart links are now recognised in their language. - **Stores whose buttons wrap their text** in another element no longer get a false "not a real button" finding. - **A sold-out product** no longer costs you points. All three changes are in AgentScore's current scoring version, 2026.10-v5. If you scanned your site before, [run AgentScore again](https://ghostagentlab.com/agentscore/) to see your updated score. ### What we took away - **Test against the real world, not only your examples.** Each bug passed every test we had, because our test sites were built the way we expected sites to be built. - **Be suspicious of good news.** The broken run would have made a better headline than the real one. A result that's surprisingly good deserves the same scrutiny as one that's surprisingly bad. - **Publish the limits.** The study states what AgentScore can't see, such as checkouts that need items in the cart and requests that don't come from AI companies' verified addresses. Saying so is what makes the rest of the numbers worth trusting. Curious how AgentScore opens and reads your pages? [How AgentScore renders pages](https://ghostagentlab.com/blog/how-agentscore-renders-pages/) explains the browser side, and [How AgentScore works](https://ghostagentlab.com/blog/how-agentscore-works/) covers the scoring. --- ## We scanned 234 online stores with AgentScore. Here's what trips up AI agents > Most online stores let AI agents in, but many trip them up once inside: unnamed buttons, missing product data, and prices agents can't see. Results from 234 stores. Published October 9, 2026 by Ghost Agent Labs · Research · https://ghostagentlab.com/blog/agentscore-store-study-2026/ AI shopping agents are starting to research and buy on people's behalf. So we asked a simple question: if an AI agent visited a typical online store today, how far would it get? We ran AgentScore on 234 online stores to find out. The short answer: most stores let agents in, but many trip them up once they're inside. **85** Median AgentScore, out of 100 **48%** of stores have buttons or links with no name an agent can read **30%** of stores don't put prices in the product page's HTML, where most agents look ### How we did it We drew 600 websites at random from the online shops listed in [Wikidata](https://www.wikidata.org/), the public knowledge base, so the sample leans toward established retailers notable enough to have an entry, from many countries. On October 9, 2026 we ran the same automated [AgentScore](https://ghostagentlab.com/agentscore/) scan (scoring version 2026.10-v5) on each, and 544 could be scanned; the rest were unreachable, timed out or returned errors. 234 of those sites were identified as stores, with products or a cart, and those are the ones in this report. Another 12 stores showed a bot challenge even to an ordinary browser request from our cloud servers, so we couldn't see them at all and left them out. For each store, AgentScore checks three things: - **Access:** whether robots.txt rules, bot protection or CAPTCHAs stop AI assistants such as ChatGPT-User, Claude-User and Perplexity-User from reaching the home page, product pages and checkout. - **Readability:** whether content, product details and prices are in the page's HTML, or only appear after JavaScript runs. - **Navigability:** whether buttons, links and form fields have names agents can read, whether pop-ups block the page, and whether the add-to-cart button is a real, uncovered button. This version doesn't send an AI agent through a live shopping task, so the scores cover whether agents can get in, read and find their way, not whether they complete a purchase. The scan is read-only: it never submits forms, adds to cart or places orders. All scans ran from a single cloud location, so pop-ups and content that vary by country may differ for visitors elsewhere. We report results only in aggregate and don't name individual stores. ### The scores The median store scored 85 out of 100. 65% scored 80 or more, and 1% scored below 50. | AgentScore | Share of stores | | --- | --- | | 0–49 | 1% | | 50–64 | 5% | | 65–79 | 29% | | 80–100 | 65% | By category, the median store earned 92% of the available points for access, 94% for readability and 75% for navigability. Getting in and reading pages is mostly solved. Finding the way around is where agents struggle. ### Finding 1: Most stores let AI assistants in The good news: robots.txt rules blocked AI assistants or AI search on only 3% of stores. Counting every way we test, 14% of stores blocked or challenged AI assistants somewhere: - **9%** blocked or challenged a request that identified itself as an AI assistant, such as ChatGPT-User or Claude-User, on the home page, while the same request as a normal browser got straight through. - **8%** kept AI assistants or AI search out of product, pricing or cart pages, through bot protection or robots.txt. - **5%** had a CAPTCHA on the home page, which agents can't solve. One caution on the bot protection figures: our requests used the AI assistants' names but came from our own servers, not from the operators' verified addresses. A site that checks where AI agents really come from would rightly turn some of these away, so these are upper bounds on how often real assistants are blocked. The robots.txt figure reads the rules each site publishes, so it isn't affected. For the stores that do block assistants, the fix is usually a settings change, not a rebuild. See [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/) and [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). ### Finding 2: Product details are where readability slips Most AI crawlers and assistant fetchers read the HTML a server sends and never run JavaScript. Here most stores do well: only 6% had less than 70% of their home page text in the HTML. Product pages are a different story: - **63%** of stores give price, currency and availability as structured data in the product page's HTML, the way agents read most reliably. - **24%** have no usable product structured data at all, so agents have to guess price and stock from the layout. - **6%** have product data that's incomplete or only in older microdata, and 7% only add it with JavaScript. - **30%** don't have prices in the product page's HTML at all. An agent that can't see a price can't compare you, and an agent that can't tell whether something is in stock will usually recommend a store where it can. See [product data AI shopping agents can read](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/). ### Finding 3: Navigation is the weakest link Navigability was the lowest-scoring category, and the most common problems in the whole study are here: - **48%** of stores have buttons or links with no name an agent can read, most often icon-only cart, search and menu buttons. - **47%** have elements that look clickable but aren't real buttons or links. - **39%** show a pop-up or banner that covers much of the page on arrival. On 28% of stores, it had no clearly labelled button to close it. - Of 118 stores where we found the add-to-cart button, **24%** had one that isn't a real button, or is covered by something else, so an agent's click may not land. These are the same problems that trip up people using screen readers, and fixing them helps both. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). ### Finding 4: Checkouts mostly let agents through AgentScore also loads each store's cart and checkout pages, without adding anything to the cart. Of 200 stores whose cart or checkout we could load, 8% blocked or challenged requests identifying as AI assistants there while browsers got through, and another 14% showed a CAPTCHA to every visitor. The same caution about verified addresses applies. Most checkouts only show their first step once something is in the cart, so we could inspect too few of them to report on guest checkout or checkout forms. See [how to make checkout work for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/) for what to look for in your own. ### Finding 5: More stores speak to agents than we expected Some stores now offer agents a direct way in. 53% publish an [llms.txt](https://ghostagentlab.com/articles/llms-txt/) file, 32% offer an MCP server or WebMCP tools, and 29% publish an agentic checkout profile. These standards are young, so we expected far lower numbers. They suggest agent-ready commerce is arriving faster than most store owners realize. ### The most common problems Across every check that covered at least 50 stores, these fell short most often (a fail or a warning): | AgentScore check | Stores falling short | | --- | --- | | Buttons and links have names | 48% | | Clickable things are real buttons and links | 47% | | llms.txt guide for AI | 47% | | Pop-ups and banners can be dismissed by agents | 39% | | Product pages give price and stock in a form agents can read | 37% | | Form fields are labelled | 34% | | Prices are in the page HTML | 30% | | Page has main and navigation landmarks | 28% | | Clear page title, description, and headings | 24% | | AI agents aren't blocked or shown a CAPTCHA at the cart and checkout | 22% | ### What the best stores do differently We compared the top quarter of stores by AgentScore with the bottom quarter. These checks showed the biggest gap in pass rates: | Check | Top quarter pass | Bottom quarter pass | | --- | --- | --- | | Product pages give price and stock in a form agents can read | 100% | 27% | | Agents can check out through an agentic commerce protocol | 72% | 0% | | llms.txt guide for AI | 86% | 16% | | Agents can use your site through MCP or WebMCP | 71% | 2% | | AI agents aren't blocked or shown a CAPTCHA at the cart and checkout | 98% | 41% | | Prices are in the page HTML | 94% | 40% | The leaders describe their products in a form agents can read, put prices in the HTML, publish an llms.txt file and offer agents a direct way in. Several of these are a one-time template or settings change, which makes them the cheapest points available to most stores. ### What to fix first > 1. **Name your buttons, and make them real.** Especially icon-only cart, search and menu buttons, and the add-to-cart button. > 2. **Put product data and prices in the HTML.** Name, price, currency and availability as structured data, rendered on the server. > 3. **Get pop-ups out of the way** on arrival, and give them a clearly labelled close button. > 4. **Check your door.** If you're among the stores that block AI assistants, let verified assistants reach your home, product and checkout pages. ### See where your store stands Run [AgentScore](https://ghostagentlab.com/agentscore/) on your own site. It's free, takes about two minutes, and shows exactly which of these problems apply to you, with a fix for each. To understand the bigger picture, start with [What is agent readiness?](https://ghostagentlab.com/articles/agent-readiness/) Method notes: scores use AgentScore scoring version 2026.10-v5. Percentages use the stores where each check could run; checks that didn't apply to a store, or couldn't be tested, are left out of that check's total. Scans ran on October 9, 2026, and sites change, so a store's score today may differ. Checks that read what buttons and links say, such as "Add to cart" or "Close", understand 19 languages, so stores in every language count toward every figure. --- ## What an AI agent sees when it visits your store > One product page, three views: the raw HTML an assistant reads, the rendered page a browser agent sees, and the accessibility tree it acts on. Published October 9, 2026 by Ghost Agent Labs · Perspective · https://ghostagentlab.com/blog/what-an-ai-agent-sees/ When your team reviews a product page, they look at it in a browser, logged in, with the cookie banner long since accepted. An AI agent arriving for a customer gets none of that. Depending on the kind of agent, it reads your raw HTML, looks at the page as it's drawn, or works through the page's accessibility tree. Here is one fictional product page seen all three ways, and what each view gets wrong. ### The page Northwind Outfitters sells running gear at `northwind.example`. Its product page for the Trail Runner 2 looks good to a person: a large photo, the name, a price of $129, a row of size swatches, an "Add to cart" button and, on a first visit, a cookie banner across the bottom half of the screen. Like many modern stores, the page is built in the browser: the server sends a shell, and JavaScript fetches the product and fills it in. A shopper asks an AI assistant: "Find me a trail running shoe under $150 in a size 10 that I can get this week." Here is what Northwind's page looks like to the agents that might come to answer. ### View 1: the raw HTML AI assistants that fetch a page because someone just asked a question, and the crawlers behind AI search, usually request the page and read the HTML that comes back. Many don't run JavaScript at all. This is what Northwind's server sends: ``` <html> <head> <title>Northwind Outfitters
Loading…
``` That's the whole story for this kind of agent. No product name, because the title is the store's name. No price, no sizes, no stock and no structured data. The assistant can't tell the shopper the Trail Runner 2 costs $129 or comes in a size 10, so it recommends a shoe from a store whose page it could read. Nothing on screen warns Northwind's team about this. The page looks perfect in every browser they own. It's the problem our guide to [JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/) is about, and the reason [Product structured data](https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/) and [machine-readable prices](https://ghostagentlab.com/articles/machine-readable-prices/) belong in the HTML the server sends. ### View 2: the rendered page Browser agents run a real browser. They load the page, let the scripts run, and decide what to do next, often by looking at a screenshot. This agent gets much more: - The photo, the name "Trail Runner 2" and the price, $129. - The size swatches, though nothing on screen says which sizes are in stock: sold-out sizes are only a lighter shade of gray. - The cookie banner, covering the bottom of the screen, including the "Add to cart" button. The agent has to deal with the banner first. If its buttons are clearly labelled "Accept" and "Reject", that's one extra step. If the only way out is a small "×" drawn in an image, the agent may click the wrong thing or give up. Either way, it costs a step on every visit where the agent starts with a fresh browser, and many do. Our guide to [cookie banners and pop-ups](https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/) covers how to keep them out of the way. ### View 3: the accessibility tree Many browser agents don't act on pixels alone. They read the page's accessibility tree: the same outline of headings, buttons, links and fields that screen readers use, with a role and a name for each item. It's compact, and it tells the agent exactly what it can press. Here is a simplified version of Northwind's: ``` banner link (no name) button (no name) main heading "Trail Runner 2" level 1 text "$129.00" generic "8" generic "9" generic "10" generic "11" button "Add to cart" dialog "We value your privacy" button "Accept all" button (no name) ``` Read it the way an agent would: - **The logo link and the cart icon have no names.** The agent can't tell which button is the cart. An `aria-label` or visible text fixes both. - **The sizes aren't controls.** They're plain boxes with a click handler, so they show up as "generic" rather than buttons or options. The agent may not realize it can choose a size at all, and nothing tells it which sizes are sold out. Real buttons or radio inputs, with the stock state in their names, solve it. See [variant pickers AI agents can use](https://ghostagentlab.com/articles/variant-pickers-ai-agents/). - **The banner's "Reject" button has no name.** It's an icon with no label. An agent that wants to decline cookies on the shopper's behalf can't find the option. The good news is in there too: one clear `h1`, a price as text, and an "Add to cart" button that is a real button with a real name. Those are the things that let an agent finish the job. ### Same page, three stories | What the agent needs | Raw HTML | Rendered page | Accessibility tree | | --- | --- | --- | --- | | Product name | Missing | Yes | Yes | | Price | Missing | Yes | Yes, as text | | Sizes in stock | Missing | Only as shading | Sizes listed, stock missing, not selectable | | A way past the banner | Not applicable | Depends on the buttons | "Accept all" only | | Add to cart | Missing | Covered by the banner | Present and named | No single view is "the" agent view. A shopper's question might be answered from the raw HTML, and the purchase made by a browser agent working from the accessibility tree. A page has to work in all three. ### How to see your own pages this way 1. **Raw HTML:** open a product page, choose "View page source", and search for the price and the product name. If they're not there, agents that read the HTML don't have them. 2. **Rendered page:** open the same page in a private window at desktop size. What covers it on arrival is what a browser agent meets first. 3. **Accessibility tree:** in Chrome's developer tools, the Accessibility pane shows each element's role and name. Look for buttons and links with no name and for clickable things that show up as "generic". [AgentScore](https://ghostagentlab.com/agentscore/) does this comparison for you. It fetches your home page and key pages as a browser and as AI assistants, renders them in a real browser, and checks names, labels, banners and the add-to-cart button. Checks such as **Content loads without JavaScript**, **Product pages give price and stock in a form agents can read**, **Buttons and links have names** and **Pop-ups and banners can be dismissed by agents** map directly onto the problems above. [How AgentScore renders your pages](https://ghostagentlab.com/blog/how-agentscore-renders-pages/) explains the details. > **For Northwind, four changes cover most of it:** render the product name, price and stock in the HTML with Product structured data; give the logo, cart icon and banner buttons readable names; make the size swatches real controls that say when a size is sold out; and keep the banner off the "Add to cart" button. None of them changes how the page looks to a person. As we've argued before, [accessibility work is agent readiness work](https://ghostagentlab.com/blog/accessibility-is-agent-readiness/). --- ## Getting agent-ready before the holiday rush > Bot rules, sale prices, stock, promo pop-ups and code freezes: what to check before peak season so AI agents can still find, price and buy. Published October 9, 2026 by Ghost Agent Labs · Guide · https://ghostagentlab.com/blog/holiday-season-agent-readiness/ Peak season is when a broken journey costs the most, and it's also when the riskiest changes go live: tighter bot rules, sale prices, promo pop-ups, then a code freeze that makes anything you missed hard to fix. Here's what to check before the freeze, so AI agents shopping for your customers can still find you, read your prices and buy. ### Why peak season is different Every change made for peak is reasonable on its own. Security tightens bot protection to stop scalpers and card testing. Marketing adds a countdown banner and an email pop-up. Merchandising loads sale prices through a promotions app. Operations shortens cache times, or lengthens them to cope with load. Then engineering freezes the code. Each of those changes is made by a different team, often in a hurry, and none of them is tested with an AI agent in mind. An assistant asked to "find a gift under $50 that arrives before the 24th" needs to get past your bot rules, read the sale price, see that the item is in stock, close the pop-up and reach the cart. Any one of those can fail quietly, and you won't see it in your analytics. ### A timeline for the weeks before the freeze | When | What to do | Who | | --- | --- | --- | | Six or more weeks out | Run AgentScore on your home page and fix anything critical. Agree which journeys matter most at peak. | E-commerce lead, developers | | Four weeks out | Review planned bot protection changes with your CDN or security team. Check sale price and stock data on a test promotion. | Security, merchandising | | Two weeks out | Put promo pop-ups and banners live on staging or behind a flag, and check an agent can still reach the cart. | Marketing, developers | | Before the freeze | Rescan, set up scheduled journey tests with alerts, and agree what counts as an exception to the freeze. | E-commerce lead | ### 1. Bot protection tightened for peak This is the change most likely to turn agents away. Common peak settings include stricter challenge thresholds, "under attack" modes that challenge every visitor, blocks on cloud and data center networks, and country blocks. Browser agents and assistants that fetch pages for people often run from cloud networks, so they get caught by rules written for scrapers. - Ask for a list of every bot rule planned for peak, and who can change them during the freeze. CDN rules often sit outside the code freeze and change mid-season. - Make sure the AI assistants you want are allowed by verified identity, not just by user agent. See [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). - Check cart and checkout separately from the home page. Many stores add extra protection there for peak. - Prefer rate limits on the expensive actions, like payment attempts, over blanket challenges. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/) and [geo-blocking and AI agents](https://ghostagentlab.com/articles/geo-blocking-ai-agents/). AgentScore checks whether your bot protection treats AI assistants differently from a browser, and whether the cart and checkout show them a block or a CAPTCHA. Rescan after every bot rule change, not just once in October. For the settings themselves, see [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/) and [setting up your CDN for AI agents](https://ghostagentlab.com/articles/cdn-settings-ai-agents/). ### 2. Sale prices that agents can read Promotion tools often change the price in the browser with JavaScript after the page loads. A shopper sees the sale price. An agent that reads the HTML sees the full price, or no price at all, and may quote the wrong figure to its user or skip you. Before peak, run a test promotion and check three places: the visible price in the page source (use View Source, not the inspector), the price in your product structured data, and the price in the cart. They should all agree. If a sale ends on a known date, say so: ``` "offers": { "@type": "Offer", "price": "39.00", "priceCurrency": "USD", "priceValidUntil": "2026-12-01", "availability": "https://schema.org/InStock" } ``` Show the original and the sale price in text, such as "Was $59, now $39", rather than relying on a strikethrough alone. See [prices AI agents can read](https://ghostagentlab.com/articles/machine-readable-prices/). ### 3. Stock that stays accurate At peak, stock changes by the hour. If your structured data says `InStock` after an item sells out, an assistant will send people to a product they can't buy. If it says `OutOfStock` for an item you restocked this morning, you lose the recommendation. - Make sure `availability` in your product data comes from the same source as the "Add to cart" button, not a nightly export. - Use `PreOrder` or `BackOrder` where they apply, and state delivery estimates in text. - If you lengthen CDN cache times for load, check how stale product pages can get, and purge on stock changes for best sellers. ### 4. Promo pop-ups and banners Email capture, spin-to-win wheels, countdown banners and free-shipping bars all multiply in November. An agent can only get past one if it has a real, labelled close button and doesn't cover the content behind it. A pop-up that sits over the "Add to cart" button means the agent's click lands on the pop-up instead. AgentScore checks that pop-ups and banners can be dismissed, and whether the add-to-cart button on a product page is covered by something else. Check again once peak promotions are live, because they're usually added after the last scan. See [cookie banners and pop-ups](https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/). ### 5. Promo codes, carts and checkout Gift shopping by agent depends on the last few steps working. Make sure the promo code field is labelled, that a rejected code produces a clear error in text, and that the cart total updates in the page. See [carts AI agents can manage](https://ghostagentlab.com/articles/cart-ai-agents/) and [error messages AI agents can understand](https://ghostagentlab.com/articles/error-messages-ai-agents/). Resist adding a CAPTCHA to checkout for peak. It stops agents at the very last step, after they've done all the work. Card testing is better handled by your payment provider's fraud tools and by rate limits on payment attempts. Keep guest checkout on. See [checkout for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/) and [guest checkout](https://ghostagentlab.com/articles/guest-checkout-ai-agents/). ### 6. Holiday policies in plain text Assistants are often asked about delivery cut-off dates and extended returns. Put them on your shipping and returns pages as text in the HTML, with dates, not only in a banner image or a pop-up. See [policy pages AI assistants can read](https://ghostagentlab.com/articles/policy-pages-ai-assistants/). ### Planning for the freeze itself A code freeze protects you from risky changes, but it also means a problem found in December may wait until January. Three things help: 1. **Agree that blocking AI agents counts as an incident.** If an assistant can't reach your product pages or checkout, that should qualify for a freeze exception, the same as a broken payment form. 2. **Test the money journeys on a schedule.** [Ghost Agents](https://ghostagentlab.com/ghost-agent/) run real journeys, such as search, choose a size and add to cart, stop at the payment form, and alert you when a journey that used to pass starts failing, with a step-by-step replay. 3. **Watch your logs, not just analytics.** Most agents don't run your analytics tag. A rise in blocked or failed agent requests shows up in server and CDN logs first. See [finding the errors AI agents hit](https://ghostagentlab.com/articles/agent-errors-in-logs/). > **Before the freeze, check:** > > - Every peak bot rule has been reviewed, and verified AI assistants still get through to product pages, cart and checkout. > - Sale prices appear in the page source and in structured data, and match the cart. > - Stock status in product data updates when items sell out. > - Every promo pop-up has a labelled close button and doesn't cover "Add to cart". > - No new CAPTCHA at checkout. Guest checkout is on. > - Delivery cut-offs and holiday returns are written out on your policy pages. > - Scheduled journey tests and alerts go to someone who can act during the freeze. Start with a fresh [AgentScore](https://ghostagentlab.com/agentscore/) scan, then work through the list. If you have more time, our [90-day agent readiness plan](https://ghostagentlab.com/blog/90-day-agent-readiness-plan/) covers the rest. --- ## Agent readiness for software companies: pricing, sign-up and docs > For software companies, the pricing page is the product page and sign-up is the checkout. How to get pricing, trials, docs, llms.txt and MCP right. Published October 9, 2026 by Ghost Agent Labs · Perspective · https://ghostagentlab.com/blog/agent-readiness-for-saas/ Most agent readiness advice is written for online stores. But software buyers send AI agents to do research too: compare plans, check whether a tool integrates with what they already use, find the limits of a free tier, and start a trial. For a software company, the pricing page is the product page, the sign-up form is the checkout, and the docs are where the real evaluation happens. ### How AgentScore sees a software site AgentScore looks for the pages an agent acting for a customer would need, the way an agent would: links in your home page's HTML, then your sitemap, then common addresses such as `/pricing`. If it finds product pages or a cart, it treats the site as a store. If it finds a pricing page and no products or cart, it treats the site as software ("saas" in the scan's data). That changes what some checks look for. | Check | On a software site, it looks at | | --- | --- | | AI agents can open your product, pricing and cart pages | Whether AI assistants can load your pricing page, not just a browser | | Prices are in the page HTML | Whether plan prices are in the HTML your server sends, before JavaScript runs | | Key pages are linked from the home page | Whether pricing is linked from the home page with an ordinary link | | Agents can find and press your add-to-cart or sign-up button | Whether the pricing page has a real, visible button such as "Start free trial" or "Buy Pro" | | Cart, checkout and agentic commerce checks | Nothing. They're marked "not tested", so they don't count against you | Everything else, from robots.txt and bot protection to structured data, named buttons and labelled form fields, applies the same way as for any site. See [what is agent readiness?](https://ghostagentlab.com/articles/agent-readiness/) for the full model. ### Pricing pages agents can read Pricing pages are often the most designed page on a software site, and that's where agents struggle. Common problems: - **Prices loaded by JavaScript.** A monthly/annual toggle that fetches prices when clicked leaves the HTML with no prices at all. Put both sets of prices in the HTML and let the toggle show and hide them. - **Features shown only as icons.** A comparison table of tick and cross images says nothing to an agent unless each icon has a text label, such as "Included" or "Not included". See [image alt text](https://ghostagentlab.com/articles/image-alt-text-ai-agents/). - **Units left implicit.** Say "per user per month, billed annually" in text, not in a tooltip. Agents compare plans across vendors, and unclear units make you look more expensive or less clear than a competitor. - **"Contact sales" with no explanation.** If a plan is priced on request, say so in plain text and say what affects the price. An agent can then tell its user how to get a quote instead of guessing. A plain HTML table is still the clearest way to compare plans: ```
Northwind plans, billed annually
StarterPro
Price$12 per user per month$29 per user per month
Single sign-onNot includedIncluded
``` For more, see [prices AI agents can read](https://ghostagentlab.com/articles/machine-readable-prices/). ### Sign-up and trials An agent that has chosen your product for someone will try to start the trial. It needs a visible, real button with a clear label, then a form it can fill in: every field labelled, errors explained in text, and no surprises. See [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/). Some steps should stay with the person, and that's fine. Email verification, single sign-on, payment details for a card-required trial and accepting terms are all reasonable places for an agent to hand back to its user. The aim is for the agent to get as far as it should, then stop cleanly with a clear message, rather than fail at a CAPTCHA on the first screen. See [sign-up, booking and lead forms AI agents can finish](https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/). [Ghost Agents](https://ghostagentlab.com/ghost-agent/) can test this journey for real: from the home page to pricing to the sign-up form, using test data you supply, with a step-by-step replay showing where an agent got stuck. ### Docs are your product page for agents For many software products, the docs decide the sale. Coding assistants and research agents read them to answer "does it support X?" and "how hard is it to set up?". Treat docs with the same care as marketing pages: - Serve docs as HTML that works without JavaScript. Some docs sites are single-page apps that send an empty shell. See [JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). - Keep public docs public. If setup guides sit behind a login, agents can't use them to recommend you. See [login walls and gated content](https://ghostagentlab.com/articles/login-walls-ai-agents/). - Give each version a stable address and point old copies at the current one, so agents don't quote outdated instructions. See [canonical URLs](https://ghostagentlab.com/articles/canonical-urls-ai-agents/). - Publish limits, supported integrations and security information as text, not only in PDFs or sales decks. ### llms.txt for software companies [llms.txt](https://llmstxt.org/) is a proposed convention, not a standard, and AI providers don't all say whether they read it. It's cheap to publish, though, and it suits software companies well, because your most useful pages are easy to list. Some docs platforms can generate one for you. ``` # Northwind > Northwind is scheduling software for field service teams. ## Product - [Pricing](https://northwind.example/pricing): plans, limits and what each includes - [Start a free trial](https://northwind.example/signup): 14 days, no card required ## Docs - [Getting started](https://northwind.example/docs/start) - [API reference](https://northwind.example/docs/api) - [Integrations](https://northwind.example/integrations) ## Trust - [Security](https://northwind.example/security) - [Status](https://status.northwind.example/) ``` AgentScore checks that an llms.txt file exists. It can't tell whether yours is up to date, so add it to your release checklist. See [how to write an llms.txt file](https://ghostagentlab.com/articles/llms-txt/). ### MCP: your product, not just your website For a software company, the [Model Context Protocol](https://modelcontextprotocol.io/) is about more than your website. An MCP server can let an assistant use your product itself on a customer's behalf: look up records, create tasks, run reports. That's product work, with sign-in and permissions to design carefully, and many software companies are already building one. There's no single agreed way to advertise an MCP server on a website yet. AgentScore's "Agents can use your site through MCP or WebMCP" check looks for a server card at `/.well-known/mcp.json`, an MCP link in the page, a mention in llms.txt, or WebMCP tools in the page. If you have a server, mentioning it in llms.txt and your docs is the simplest start. See [MCP and WebMCP for websites](https://ghostagentlab.com/articles/mcp-webmcp-for-websites/). ### Where to start 1. Run [AgentScore](https://ghostagentlab.com/agentscore/) on your home page. If "Key pages are linked from the home page" doesn't pass, agents may not be finding your pricing page. 2. View the source of your pricing page and check every price and plan feature is there as text. 3. Make "Start free trial" a real, clearly labelled button, and check the sign-up form's fields are labelled. 4. Publish an llms.txt that links to pricing, sign-up, docs and your trust pages. 5. Test the journey from home page to sign-up with a [Ghost Agent](https://ghostagentlab.com/ghost-agent/). --- ## How to explain agent readiness to your board > A one-page way to explain agent readiness to your board: what's changing, the risks, what to measure and what to ask for, without hype or invented numbers. Published October 9, 2026 by Ghost Agent Labs · Guide · https://ghostagentlab.com/blog/explain-agent-readiness-to-your-board/ Your board doesn't need to know what robots.txt is. It needs to know what's changing, what that risks for the business, what you're doing about it, and how it will know the work is paying off. Here's a way to say all of that on one page, without hype and without numbers you can't stand behind. ### The one-page version Start with this structure. Each heading is one or two sentences. Fill in the bracketed parts with your own findings. > **What's changing.** People are starting to ask AI assistants to research, compare and buy on their behalf. Those assistants visit our website the way a customer would, but they read it differently. > > **Why it matters to us.** If an assistant can't get in, can't read our prices or can't complete a purchase, it recommends someone else, and we never see the lost customer in our reports. > > **Where we stand.** \[Our AgentScore, the main issues it found, and whether AI agents can complete our most important journey today.\] > > **What we're doing.** \[Three to five fixes, who owns each, and when they'll be done.\] > > **What we need.** \[Time, tools or decisions, each with a review date.\] That's it. Everything below is about filling it in well. ### Explaining the shift without hype Boards have heard a lot about AI. The quickest way to lose the room is a slide of large market forecasts. You don't need them. The argument for agent readiness is modest and easy to defend: - AI agents already visit websites for people. You can show this from your own server or CDN logs. - Nobody knows how fast this will grow. That's a reason to be ready cheaply now, not to wait for certainty. - Much of the work also improves accessibility, search and site quality, so it isn't a bet on one trend. See [accessibility work is agent readiness work](https://ghostagentlab.com/blog/accessibility-is-agent-readiness/) and [SEO got you found. Agent readiness gets you chosen](https://ghostagentlab.com/blog/seo-to-agent-readiness/). - The failure is silent. A blocked agent doesn't complain. It goes elsewhere. That makes it the kind of risk a board should want someone to own. If you quote any outside figure, cite the source on the slide. If you can't, describe the trend in words instead. ### The risks, in board language | Risk | What it looks like | Usually owned by | | --- | --- | --- | | Lost revenue | AI assistants are blocked by bot protection, can't see prices, or fail at checkout, so they recommend or buy from a competitor. | E-commerce or digital lead | | Being misquoted | Assistants give customers the wrong price, stock or return policy because our pages are hard to read or out of date. | Marketing and content | | Impostors | Scrapers pretend to be well-known AI agents to get past our defenses. Opening the door carelessly makes this worse. | Security | | Content and legal | Decisions about AI training on our content, our terms of service, and responsibility for purchases an agent makes for a customer. | Legal, with marketing | | Blind spots | Our analytics can't see most AI agents, so decisions are made on incomplete data. | Analytics or digital lead | The point of the table is that this is a cross-team issue with no natural home. Ask the board to agree one accountable owner. See [who owns agent readiness?](https://ghostagentlab.com/blog/who-owns-agent-readiness/) ### What to measure Pick a few measures you can report every quarter, and show the trend rather than a single figure. - **AgentScore, by category.** Access, Readability and Navigability tell you whether agents can get in, understand your pages and find their way around. The free scan doesn't test Task completion, so don't present its score as proof that agents can buy. - **Journey pass rate.** Whether AI agents can complete your most important journeys, such as checkout or sign-up, from scheduled [Ghost Agent](https://ghostagentlab.com/ghost-agent/) tests. - **Agent traffic.** How many requests come from AI assistants and search agents, how many are verified, and how many are impostors, from your logs. See [how to measure agent traffic](https://ghostagentlab.com/articles/measure-ai-agent-traffic/). - **Visits and orders from AI assistants.** People who click through from an assistant's answer. See [tracking visits and sales from AI assistants](https://ghostagentlab.com/articles/ai-referral-traffic/). - **Time to fix.** How long a critical finding, such as AI agents being blocked, stays open. Be careful with attribution. You can show that more agents get through and more journeys pass. You usually can't prove exactly how much revenue that produced, and the board will trust you more if you say so. For a fuller set, see [agent readiness KPIs](https://ghostagentlab.com/articles/agent-readiness-kpis/). ### What to ask for Keep the ask small and staged, with a review point after each stage. A typical shape: 1. **A baseline.** A scan, a look at agent traffic in your logs, and one tested journey. This needs a few days of people's time, not a project. 2. **Fixes.** Developer and content time to work through the findings. Much of it overlaps with work already on the accessibility and SEO backlog, so say where it does. 3. **Ongoing testing and monitoring.** Tools that keep watching after the fixes, because a theme update or a new bot rule can undo them. 4. **Decisions.** Time from legal and security to settle policy questions such as AI training crawlers and verified agents. 5. **Optional experiments.** Newer interfaces such as [MCP and WebMCP](https://ghostagentlab.com/articles/mcp-webmcp-for-websites/) or [agentic commerce protocols](https://ghostagentlab.com/articles/agentic-commerce-protocols/), framed as small trials with a date to decide whether to continue. Our [90-day agent readiness plan](https://ghostagentlab.com/blog/90-day-agent-readiness-plan/) maps well onto the first three stages. ### Questions you'll probably get #### Are we letting AI companies take our content? No. AI training crawlers and the assistants that fetch pages for a customer use different names, so you can block one and allow the other. That's a policy choice to make once, with legal. See [should you block AI crawlers?](https://ghostagentlab.com/blog/block-or-allow-ai-crawlers/) #### Does this open us up to bots? Not if it's done properly. The goal is to let in the agents you want, checked by verified identity, and keep blocking the rest. See [how to tell if an AI crawler is real](https://ghostagentlab.com/articles/verify-ai-crawlers/). #### What happens if we do nothing? Probably nothing you'll notice, which is the problem. Some customers' assistants will fail on the site and choose a competitor, and it won't show up in any report you currently read. To fill in "Where we stand", run a free [AgentScore](https://ghostagentlab.com/agentscore/) scan. It takes under a minute and gives you a score out of 100 and a list of what to fix. --- ## Seven agent readiness mistakes to avoid > Seven common agent readiness mistakes, from blocking AI assistants with training crawlers to measuring agents with JavaScript analytics, and how to avoid each. Published October 9, 2026 by Ghost Agent Labs · Guide · https://ghostagentlab.com/blog/agent-readiness-mistakes/ Most agent readiness problems aren't decisions anyone made. They're side effects of sensible work: a security rule, a design choice, a tracking setup. Here are seven mistakes that are easy to make and easy to miss, why each one matters, and what to do instead. ### 1. Blocking AI assistants along with training crawlers Many sites decide not to let AI companies train on their content, which is a fair choice. The mistake is blocking everything with "AI" in its name, including the assistants that fetch a page because a customer asked a question right now. Those are different agents with different names: `GPTBot` collects training data, while `ChatGPT-User` and `OAI-SearchBot` fetch pages for ChatGPT users and search. Anthropic draws the same line between `ClaudeBot` and `Claude-User`. ``` # Opt out of AI training User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: / # Everyone else, including AI assistants and AI search User-agent: * Allow: / ``` **Instead:** decide on training and on assistants separately. See [should you block AI crawlers?](https://ghostagentlab.com/blog/block-or-allow-ai-crawlers/) and [robots.txt for AI agents](https://ghostagentlab.com/articles/robots-txt-ai-agents/). The same split applies to bot protection rules, which often block by category rather than by name. ### 2. Putting a CAPTCHA at checkout A CAPTCHA at checkout is often added to stop card testing, and it works on bots. It also stops every AI agent at the very last step, after it has searched, compared, chosen a size and filled the cart. That's the most expensive place to lose a customer. **Instead:** use your payment provider's fraud tools and rate limits on payment attempts, and keep challenges for traffic that is actually suspicious. AgentScore checks whether AI agents are blocked or shown a CAPTCHA at the cart and checkout. See [checkout for AI shopping agents](https://ghostagentlab.com/articles/agent-ready-checkout/) and [bot protection and CAPTCHAs](https://ghostagentlab.com/articles/bot-protection-ai-agents/). ### 3. Prices that only appear with JavaScript Many sites build product and pricing pages in the browser. A person sees the price a moment after the page loads. Many AI agents read the HTML your server sends and never run the scripts, so they see a product with no price, or a placeholder. They may skip you, or tell their user to check the site. **Instead:** make sure prices are in the HTML, through server-side rendering or static generation, and in your product structured data. Check by viewing the page source, not the browser's inspector, which shows the page after scripts run. See [prices AI agents can read](https://ghostagentlab.com/articles/machine-readable-prices/) and [JavaScript-only content](https://ghostagentlab.com/articles/javascript-content-ai-agents/). ### 4. Icons with no names A magnifying glass for search, a bag for the cart, a cross to close a pop-up. People know what they mean. Browser agents choose what to click by each control's name, the same way screen readers do, and an icon with no text or label has no name at all. To the agent, the cart button isn't there. ``` ``` **Instead:** give every icon-only button and link a label, and use real `
…
``` The last example has no role and no name. A screen reader user can't use it, and an agent reading the accessibility tree may not know it exists. ### Where WCAG and agent readiness overlap The [Web Content Accessibility Guidelines (WCAG) 2.2](https://www.w3.org/TR/WCAG22/) are the standard most accessibility programs work to. Several of their success criteria line up closely with AgentScore checks. | WCAG 2.2 success criterion | Related AgentScore check | | --- | --- | | 4.1.2 Name, Role, Value | Buttons and links have names; Clickable things are real buttons and links | | 1.3.1 Info and Relationships, 3.3.2 Labels or Instructions | Form fields are labelled; Cart and checkout fields are labelled for agents | | 1.1.1 Non-text Content | Images have text descriptions | | 2.4.1 Bypass Blocks, 1.3.1 Info and Relationships | Page has main and navigation landmarks | | 2.4.2 Page Titled, 2.4.6 Headings and Labels | Clear page title, description, and headings | | 2.1.2 No Keyboard Trap, 2.4.11 Focus Not Obscured (Minimum) | Pop-ups and banners can be dismissed by agents | The match isn't exact. WCAG asks more of each criterion than our checks test for, and our checks look at things from an agent's point of view. But the direction is the same: give every control a role and a name, label every field, describe every meaningful image, and structure the page so it can be navigated without seeing it. Our guide to [buttons, links and forms agents can use](https://ghostagentlab.com/articles/agent-friendly-buttons-forms/) covers the fixes in detail, and [alt text for AI agents](https://ghostagentlab.com/articles/image-alt-text-ai-agents/) covers product images. ### Common accessibility gaps that also stop agents - **Icon-only buttons with no label.** The cart, search and menu icons in a header are the usual culprits. Add visible text or an `aria-label`. - **Placeholder text used as a label.** It disappears when the field is filled in and isn't a reliable name. Use a real `