Agent readiness comes with a lot of new words, and some old ones used in new ways. This glossary explains the terms you'll meet in our guides, in AgentScore reports and in conversations with your developers and vendors, in plain English and with what each one means for your business.
\nAI agent. Software that uses an AI model to do something on the web for a person or a company: read a page, answer a question, compare products or complete a task. In our guides, “agent” covers everything from crawlers to agents that shop.
\nTraining crawler. A bot that collects pages to train AI models, such as GPTBot or ClaudeBot. Blocking it doesn't stop AI assistants visiting for a person.
AI search crawler. A bot that indexes pages so AI search and assistants can cite and link to them, such as OAI-SearchBot or PerplexityBot.
Assistant fetcher. An AI assistant fetching a page in real time because a person asked it something, such as ChatGPT-User or Claude-User. Each visit stands in for a customer.
Browser agent. An agent that opens your site in a real browser and clicks, types and scrolls like a person, to book, sign up or buy. Also called a computer-use agent or autonomous agent.
\nUser agent. The name a visitor sends with every request to say what it is, such as a browser version or GPTBot. Easy to fake, so it shouldn't be trusted on its own.
For how these differ and how to treat each, see the AI agents that visit your website.
\nrobots.txt. A file at the root of your site that tells crawlers and agents which pages they may visit. Well-behaved agents follow it; scrapers ignore it. See robots.txt for AI agents.
\nBot protection. Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors. Set too strictly, they turn away AI assistants as well as scrapers. See bot protection and CAPTCHAs.
\nCDN. Content delivery network: the service that sits in front of your site and delivers it quickly around the world. It's often where bot rules live, and where agent traffic is recorded. See setting up your CDN for AI agents.
\nCAPTCHA or challenge. A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page. Agents can't pass them, so one on arrival or at checkout ends the visit.
\nRate limit. A cap on how many requests a visitor can make in a given time. Fair limits stop scrapers without turning away agents. See rate limits for AI agents.
\nVerification. Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature. See how to tell if an AI crawler is real.
\nImpostor (spoofed agent). A request that uses a trusted agent's name but fails verification. Usually a scraper trying to get past bot protection.
\nForward-confirmed reverse DNS. A verification check that an IP address's hostname belongs to the operator, and that the hostname points back to the same address.
\nWeb Bot Auth. A new standard in which agents sign their requests cryptographically, so sites can prove who sent them. Our own scanner and Ghost Agents sign theirs. See why we sign every request.
\nllms.txt. A short Markdown file at /llms.txt that tells AI what your business does and links to your key pages. An emerging standard. See how to write an llms.txt file.
Guest checkout. Letting shoppers buy without creating an account. An agent buying for someone can't easily sign up for them. See guest checkout.
\nRaw HTML. The page exactly as your server sends it, before any JavaScript runs. It's all that many crawlers and assistant fetchers ever see. To look at it, use your browser's “View page source”, not the inspector.
\nServer-side rendering (SSR) and static generation. Ways of building pages so the content is already in the raw HTML, rather than added afterwards by JavaScript. See why AI agents can't see JavaScript-only content.
\nStructured data (schema.org, JSON-LD). Machine-readable facts in your page code, such as a product's name, price and stock, written in the shared schema.org vocabulary, usually as JSON-LD. Agents can read it without guessing. See product data AI shopping agents can read.
\nXML sitemap. A file listing the pages you want found, so crawlers and agents don't depend on following links. See XML sitemaps for AI agents.
\nCanonical URL. The one address you name as the main version of a page that can be reached in several ways. It stops agents treating duplicates as different products. See canonical URLs.
\nAlt text. A short text description of an image. Agents can't see pictures, so this is how they learn what a product photo shows. See alt text for AI agents.
\nAccessibility tree. The structured view of a page that screen readers and many browser agents use: its headings, links, buttons and form fields, with their names. If something isn't in it, many agents can't use it. See what an AI agent sees.
\nAccessible name. The name a button, link or field has in the accessibility tree, from its text, label or aria-label. An icon-only button with no name is invisible to agents.
Landmarks. Labels for the main regions of a page, such as header, navigation, main content and footer, that let agents skip straight to what they need.
\nOverlay. Anything that covers the page, such as a cookie banner, newsletter pop-up or chat window. If an agent can't close it, the journey ends there. See cookie banners and pop-ups.
\nWCAG. The Web Content Accessibility Guidelines, the standard for making sites usable by people with disabilities. Much of it helps agents too. See accessibility work is agent readiness work.
\nJourney. A task a visitor sets out to complete on your site, such as finding a product and reaching checkout, or booking a call. The journeys that make you money are the ones to test first.
\nTask completion. Whether an agent can actually finish a journey. It's the AgentScore category worth the most points, and the only way to measure it is to try. See how to test your key journeys.
\nMCP (Model Context Protocol). A standard way to give AI agents direct tools for your site, such as search, product details and cart, instead of making them click through pages.
\nWebMCP. A proposal for offering those same tools from inside your web pages, so a browser agent can call them while it's on your site. See MCP and WebMCP.
\nAgentic commerce. Purchases that an AI agent makes for a shopper. Protocols such as the Agentic Commerce Protocol, the Universal Commerce Protocol and AP2 aim to make that safe for buyer, store and payment provider. See agentic commerce protocols explained.
\nAgent readiness. How well AI agents can get into your site, understand it, find their way around it and complete tasks on it. See what is agent readiness?
\nAgentScore. Our free score out of 100 for agent readiness, across Access (25 points), Readability (25), Navigability (20) and Task completion (30). The free scan marks Task completion “not tested” and scores the other three. See how AgentScore works.
\nCheck. One thing AgentScore tests, such as “Buttons and links have names”, reported as a pass, warning or fail, with who usually fixes it.
\nGhost Agent. A real AI agent we send through a journey on your site, on a schedule, to see whether agents can complete it. You get a step-by-step replay of each run, and an alert when a journey breaks. See Ghost Agent.
\nRun and pass rate. A run is one attempt by one agent at a journey. AI agents don't behave the same way every time, so journeys are tried several times, and the pass rate is the share of runs that succeed.
\nFlaky. A journey where some runs pass and some fail. Agents can do it, but not reliably, which usually points to a slow page, a pop-up that appears sometimes, or an unclear control.
\nAgent traffic. Visits from AI agents of every kind. Most never run your analytics tag, so it's measured from server or CDN logs. See how to measure agent traffic.
\nAI referral. A person who arrives at your site by clicking a link in an AI assistant's answer. Unlike agent visits, these do show up in analytics. See tracking visits and sales from AI assistants.
\nCrawls per AI referral. How many pages an AI company's agents fetched for every visitor its assistant sent you. A low number means you get traffic back for what they read.
\nTo see where your own site stands on each of these, run AgentScore. It's free and takes under a minute.
", "date_published": "2026-10-09T13:26:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Start here" ] }, { "id": "https://ghostagentlab.com/articles/kinds-of-ai-agents/", "url": "https://ghostagentlab.com/articles/kinds-of-ai-agents/", "title": "The AI agents that visit your website, and what each one wants", "summary": "Training crawlers, AI search crawlers, assistant fetchers, browser agents and tool-using agents: what each does, what it needs from your site, and how to treat it.", "content_html": "“AI bots” isn't one kind of visitor. Some collect pages to train models, some build AI search, some fetch a page because a person just asked about you, and some open your site in a browser and click through it. This guide explains the five kinds, what each one needs from your site, and how to treat each.
\nEach kind of agent does a different job for a different company, and brings you a different kind of value. A training crawler takes your content and may never send anyone back. An assistant fetching your returns policy is answering a customer's question right now. A browser agent on your checkout page is trying to place an order.
\nTreat them all as “bots” and you get one of two bad outcomes: block everything and disappear from AI answers and purchases, or allow everything without knowing what you've agreed to. The rest of this guide gives you the vocabulary to make a separate decision for each.
\n| Kind | Why it visits | How it reads your site | What it's worth to you |
|---|---|---|---|
| Training crawler | To collect pages for training AI models | Raw HTML, in bulk | Indirect at best |
| AI search crawler | To index pages so AI search can cite and link to them | Mostly raw HTML | Visibility in AI answers |
| Assistant fetcher | A person asked an assistant about a specific page or question | Usually raw HTML, one page at a time | A customer asking about you right now |
| Browser agent | A person asked an assistant to do something, like book or buy | A real browser: renders the page, clicks and types | A task, often a sale, in progress |
| Tool-using agent | To call tools your site offers directly, through MCP or an API | Structured answers from your tools, not pages | The fastest, most reliable route to a task |
Training crawlers collect large numbers of pages to train AI models. Examples include GPTBot, ClaudeBot and CCBot. They don't act for a particular person, and a visit from one doesn't mean anyone is looking for you.
AI search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, build the index that AI search and assistants draw on when they answer questions and cite sources. They work much like Googlebot does for traditional search.
When someone asks ChatGPT, Claude or Perplexity “what's the return window at this store?” or “compare these two products”, the assistant may fetch the pages it needs there and then. Those requests use names like ChatGPT-User, Claude-User and Perplexity-User.
Browser agents go further: they open your site in a real browser and use it the way a person would, to search, choose a size, fill in a form or reach checkout. Several AI assistants now offer an agent mode that does this on a person's behalf, and businesses are starting to use them too.
\nInstead of reading pages, some agents call tools a site offers them directly: “search products”, “check stock”, “add to cart”. These tools are offered through the Model Context Protocol (MCP), WebMCP in the browser, or agentic commerce protocols for checkout.
\nYour web analytics probably shows almost none of this. Most agents never run the JavaScript tag analytics depends on, and the ones that do are often filtered out as bots. To see them, look at your server or CDN logs, where every request is recorded with its user agent and IP address. Then check that the visitors claiming to be well-known agents really are: names are easy to fake.
\nGhost Agent Labs does this for you from your CDN or server logs. See how to measure agent traffic.
\nFor a fuller decision guide by business type, see should you block AI crawlers? To see how your site treats each kind today, run AgentScore: its robots.txt check reads your rules as 21 AI agents would.
", "date_published": "2026-10-09T13:25:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Start here" ] }, { "id": "https://ghostagentlab.com/blog/what-scanning-600-stores-taught-us/", "url": "https://ghostagentlab.com/blog/what-scanning-600-stores-taught-us/", "title": "What scanning 600 stores taught us about our own scanner", "summary": "Running AgentScore on 600 real stores exposed three bugs in our own scanner, and one run that looked perfect because almost nothing was tested. What we found, fixed and changed.", "content_html": "To write our study of online stores, we ran AgentScore on 600 real websites in one day. The study taught us a lot about stores. It taught us even more about our own scanner: three bugs that tests on our own fixture sites never caught, and one run whose results looked perfect because almost nothing had been tested. Here's what we found, what we changed, and what it means for your score.
\nEvery AgentScore check is tested against small, carefully built example sites. That proves the check does what we meant. It doesn't prove we meant the right thing for the thousands of ways real stores are built. Scanning 600 stores, from many countries, platforms and sizes, was the first time AgentScore met that variety all at once, and the first time we read its results in aggregate, where a check that's wrong for many stores stands out.
\nSeveral checks work by reading what a button or link says: finding the add-to-cart button, recognising the button that closes a cookie banner, spotting the link to the cart. All of them only knew English words. On a German store, \"In den Warenkorb\" wasn't recognised as an add-to-cart button, and \"Alle akzeptieren\" didn't count as a way to close the banner.
\nIn our first run, 61 stores whose pages weren't in English were reported as having no add-to-cart button at all. When we rescanned eight of them with the fix, AgentScore found the button on six. The other two turned out to be a different problem (see bug 3).
\nThere was a subtler issue underneath. The usual way to match whole words in code only understands the 26 letters of the English alphabet, so even a correctly translated phrase could fail to match at the edges of words in Russian or with accented letters. Japanese, Chinese and Korean don't put spaces between words at all.
\nWhat we changed: the words AgentScore looks for now live in one list covering 19 languages, including German, French, Spanish, Italian, Dutch, the Nordic languages, Polish, Turkish, Russian, Ukrainian, Japanese, Chinese and Korean, and matching works in every alphabet. It also changed which sites AgentScore recognises as stores, because finding the cart link is part of that: the same sample produced 230 stores instead of 219.
\nAgentScore checks that your add-to-cart button is a real button, because agents look for buttons and links, not for text that happens to be clickable. To find the button, it looked for the smallest element whose text matched. Many shop themes write their buttons like this:
\n<button type=\"submit\">\n <span>Add to cart</span>\n</button>The smallest element containing \"Add to cart\" is the <span>, so AgentScore reported \"a plain <span>, not a real button\", on a perfectly good button. That's a false alarm, and an expensive one: in the study's second run, 57% of the stores where we found the button were flagged. After the fix, the true figure is 24%.
What we changed: when the matching text sits inside a real button or link, AgentScore now judges that button or link. A <div> styled to look like a button is still caught, because that really is a problem for agents.
AgentScore checks one product page per store. When that product was sold out, its buy button had been replaced by a disabled \"Sold out\" button, \"売り切れ\" on a Japanese store or \"Agotado\" on a Spanish one, and AgentScore reported that it couldn't find an add-to-cart button. That says nothing about the store's buttons, only about its stock.
\nWhat we changed: AgentScore now recognises a sold-out product, from its button in any of the 19 languages or from the product data saying it's out of stock. Then it does what a shopper would: it tries up to two other products. If they're all sold out, the check is marked as not tested, which never costs points. In the study, AgentScore moved past a sold-out product on 4 stores and skipped the check on 5 where everything it tried was sold out.
\nOne of our runs came back looking wonderful. Navigation scored 100%. Not one store had an unnamed button. No pop-ups blocked anything.
\nIt was wrong. After an update to how AgentScore opens pages in its browser, the browser on our scanning machine couldn't complete secure connections, and it couldn't load a single page. Every check that needs a real browser was skipped, on 245 of 245 stores. Skipped checks don't cost points, by design, so the scores went up instead of down.
\nWe caught it because the numbers were too good to be true, not because anything failed. That's not good enough, so we added a safeguard: our analysis now refuses to produce results if more than 10% of stores couldn't be opened in a browser. We fixed the scanning machine's setup, confirmed one scan by hand, and ran all 600 sites again.
\nYour own scans are protected the same way. When AgentScore can't open your pages in a browser, your report says so and marks those checks as not tested. It never presents a skipped check as a pass. See how to read your AgentScore report.
\nAll three changes are in AgentScore's current scoring version, 2026.10-v5. If you scanned your site before, run AgentScore again to see your updated score.
\nCurious how AgentScore opens and reads your pages? How AgentScore renders pages explains the browser side, and How AgentScore works covers the scoring.
", "date_published": "2026-10-09T13:24:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Engineering" ] }, { "id": "https://ghostagentlab.com/blog/agentscore-store-study-2026/", "url": "https://ghostagentlab.com/blog/agentscore-store-study-2026/", "title": "We scanned 234 online stores with AgentScore. Here's what trips up AI agents", "summary": "Most online stores let AI agents in, but many trip them up once inside: unnamed buttons, missing product data, and prices agents can't see. Results from 234 stores.", "content_html": "AI shopping agents are starting to research and buy on people's behalf. So we asked a simple question: if an AI agent visited a typical online store today, how far would it get? We ran AgentScore on 234 online stores to find out. The short answer: most stores let agents in, but many trip them up once they're inside.
\nMedian AgentScore, out of 100
of stores have buttons or links with no name an agent can read
of stores don't put prices in the product page's HTML, where most agents look
We drew 600 websites at random from the online shops listed in Wikidata, the public knowledge base, so the sample leans toward established retailers notable enough to have an entry, from many countries. On October 9, 2026 we ran the same automated AgentScore scan (scoring version 2026.10-v5) on each, and 544 could be scanned; the rest were unreachable, timed out or returned errors. 234 of those sites were identified as stores, with products or a cart, and those are the ones in this report. Another 12 stores showed a bot challenge even to an ordinary browser request from our cloud servers, so we couldn't see them at all and left them out.
\nFor each store, AgentScore checks three things:
\nThis version doesn't send an AI agent through a live shopping task, so the scores cover whether agents can get in, read and find their way, not whether they complete a purchase. The scan is read-only: it never submits forms, adds to cart or places orders. All scans ran from a single cloud location, so pop-ups and content that vary by country may differ for visitors elsewhere. We report results only in aggregate and don't name individual stores.
\nThe median store scored 85 out of 100. 65% scored 80 or more, and 1% scored below 50.
\n| AgentScore | Share of stores |
|---|---|
| 0–49 | 1% |
| 50–64 | 5% |
| 65–79 | 29% |
| 80–100 | 65% |
By category, the median store earned 92% of the available points for access, 94% for readability and 75% for navigability. Getting in and reading pages is mostly solved. Finding the way around is where agents struggle.
\nThe good news: robots.txt rules blocked AI assistants or AI search on only 3% of stores. Counting every way we test, 14% of stores blocked or challenged AI assistants somewhere:
\nOne caution on the bot protection figures: our requests used the AI assistants' names but came from our own servers, not from the operators' verified addresses. A site that checks where AI agents really come from would rightly turn some of these away, so these are upper bounds on how often real assistants are blocked. The robots.txt figure reads the rules each site publishes, so it isn't affected.
\nFor the stores that do block assistants, the fix is usually a settings change, not a rebuild. See bot protection and CAPTCHAs and robots.txt for AI agents.
\nMost AI crawlers and assistant fetchers read the HTML a server sends and never run JavaScript. Here most stores do well: only 6% had less than 70% of their home page text in the HTML.
\nProduct pages are a different story:
\nAn agent that can't see a price can't compare you, and an agent that can't tell whether something is in stock will usually recommend a store where it can. See product data AI shopping agents can read.
\nNavigability was the lowest-scoring category, and the most common problems in the whole study are here:
\nThese are the same problems that trip up people using screen readers, and fixing them helps both. See buttons, links and forms agents can use.
\nAgentScore also loads each store's cart and checkout pages, without adding anything to the cart. Of 200 stores whose cart or checkout we could load, 8% blocked or challenged requests identifying as AI assistants there while browsers got through, and another 14% showed a CAPTCHA to every visitor. The same caution about verified addresses applies.
\nMost checkouts only show their first step once something is in the cart, so we could inspect too few of them to report on guest checkout or checkout forms. See how to make checkout work for AI shopping agents for what to look for in your own.
\nSome stores now offer agents a direct way in. 53% publish an llms.txt file, 32% offer an MCP server or WebMCP tools, and 29% publish an agentic checkout profile. These standards are young, so we expected far lower numbers. They suggest agent-ready commerce is arriving faster than most store owners realize.
\nAcross every check that covered at least 50 stores, these fell short most often (a fail or a warning):
\n| AgentScore check | Stores falling short |
|---|---|
| Buttons and links have names | 48% |
| Clickable things are real buttons and links | 47% |
| llms.txt guide for AI | 47% |
| Pop-ups and banners can be dismissed by agents | 39% |
| Product pages give price and stock in a form agents can read | 37% |
| Form fields are labelled | 34% |
| Prices are in the page HTML | 30% |
| Page has main and navigation landmarks | 28% |
| Clear page title, description, and headings | 24% |
| AI agents aren't blocked or shown a CAPTCHA at the cart and checkout | 22% |
We compared the top quarter of stores by AgentScore with the bottom quarter. These checks showed the biggest gap in pass rates:
\n| Check | Top quarter pass | Bottom quarter pass |
|---|---|---|
| Product pages give price and stock in a form agents can read | 100% | 27% |
| Agents can check out through an agentic commerce protocol | 72% | 0% |
| llms.txt guide for AI | 86% | 16% |
| Agents can use your site through MCP or WebMCP | 71% | 2% |
| AI agents aren't blocked or shown a CAPTCHA at the cart and checkout | 98% | 41% |
| Prices are in the page HTML | 94% | 40% |
The leaders describe their products in a form agents can read, put prices in the HTML, publish an llms.txt file and offer agents a direct way in. Several of these are a one-time template or settings change, which makes them the cheapest points available to most stores.
\nRun AgentScore on your own site. It's free, takes about two minutes, and shows exactly which of these problems apply to you, with a fix for each. To understand the bigger picture, start with What is agent readiness?
\nMethod notes: scores use AgentScore scoring version 2026.10-v5. Percentages use the stores where each check could run; checks that didn't apply to a store, or couldn't be tested, are left out of that check's total. Scans ran on October 9, 2026, and sites change, so a store's score today may differ. Checks that read what buttons and links say, such as \"Add to cart\" or \"Close\", understand 19 languages, so stores in every language count toward every figure.
", "date_published": "2026-10-09T13:23:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Research" ] }, { "id": "https://ghostagentlab.com/blog/what-an-ai-agent-sees/", "url": "https://ghostagentlab.com/blog/what-an-ai-agent-sees/", "title": "What an AI agent sees when it visits your store", "summary": "One product page, three views: the raw HTML an assistant reads, the rendered page a browser agent sees, and the accessibility tree it acts on.", "content_html": "When your team reviews a product page, they look at it in a browser, logged in, with the cookie banner long since accepted. An AI agent arriving for a customer gets none of that. Depending on the kind of agent, it reads your raw HTML, looks at the page as it's drawn, or works through the page's accessibility tree. Here is one fictional product page seen all three ways, and what each view gets wrong.
\nNorthwind Outfitters sells running gear at northwind.example. Its product page for the Trail Runner 2 looks good to a person: a large photo, the name, a price of $129, a row of size swatches, an \"Add to cart\" button and, on a first visit, a cookie banner across the bottom half of the screen. Like many modern stores, the page is built in the browser: the server sends a shell, and JavaScript fetches the product and fills it in.
A shopper asks an AI assistant: \"Find me a trail running shoe under $150 in a size 10 that I can get this week.\" Here is what Northwind's page looks like to the agents that might come to answer.
\nAI assistants that fetch a page because someone just asked a question, and the crawlers behind AI search, usually request the page and read the HTML that comes back. Many don't run JavaScript at all. This is what Northwind's server sends:
\n<html>\n<head>\n <title>Northwind Outfitters</title>\n</head>\n<body>\n <header><a href=\"/\"><img src=\"/logo.svg\"></a></header>\n <div id=\"app\">Loading…</div>\n <script src=\"/assets/app.4f9c.js\"></script>\n</body>\n</html>That's the whole story for this kind of agent. No product name, because the title is the store's name. No price, no sizes, no stock and no structured data. The assistant can't tell the shopper the Trail Runner 2 costs $129 or comes in a size 10, so it recommends a shoe from a store whose page it could read.
\nNothing on screen warns Northwind's team about this. The page looks perfect in every browser they own. It's the problem our guide to JavaScript-only content is about, and the reason Product structured data and machine-readable prices belong in the HTML the server sends.
\nBrowser agents run a real browser. They load the page, let the scripts run, and decide what to do next, often by looking at a screenshot. This agent gets much more:
\nThe agent has to deal with the banner first. If its buttons are clearly labelled \"Accept\" and \"Reject\", that's one extra step. If the only way out is a small \"×\" drawn in an image, the agent may click the wrong thing or give up. Either way, it costs a step on every visit where the agent starts with a fresh browser, and many do. Our guide to cookie banners and pop-ups covers how to keep them out of the way.
\nMany browser agents don't act on pixels alone. They read the page's accessibility tree: the same outline of headings, buttons, links and fields that screen readers use, with a role and a name for each item. It's compact, and it tells the agent exactly what it can press. Here is a simplified version of Northwind's:
\nbanner\n link (no name)\n button (no name)\nmain\n heading \"Trail Runner 2\" level 1\n text \"$129.00\"\n generic \"8\"\n generic \"9\"\n generic \"10\"\n generic \"11\"\n button \"Add to cart\"\ndialog \"We value your privacy\"\n button \"Accept all\"\n button (no name)Read it the way an agent would:
\naria-label or visible text fixes both.The good news is in there too: one clear h1, a price as text, and an \"Add to cart\" button that is a real button with a real name. Those are the things that let an agent finish the job.
| What the agent needs | Raw HTML | Rendered page | Accessibility tree |
|---|---|---|---|
| Product name | Missing | Yes | Yes |
| Price | Missing | Yes | Yes, as text |
| Sizes in stock | Missing | Only as shading | Sizes listed, stock missing, not selectable |
| A way past the banner | Not applicable | Depends on the buttons | \"Accept all\" only |
| Add to cart | Missing | Covered by the banner | Present and named |
No single view is \"the\" agent view. A shopper's question might be answered from the raw HTML, and the purchase made by a browser agent working from the accessibility tree. A page has to work in all three.
\nAgentScore does this comparison for you. It fetches your home page and key pages as a browser and as AI assistants, renders them in a real browser, and checks names, labels, banners and the add-to-cart button. Checks such as Content loads without JavaScript, Product pages give price and stock in a form agents can read, Buttons and links have names and Pop-ups and banners can be dismissed by agents map directly onto the problems above. How AgentScore renders your pages explains the details.
\nFor Northwind, four changes cover most of it: render the product name, price and stock in the HTML with Product structured data; give the logo, cart icon and banner buttons readable names; make the size swatches real controls that say when a size is sold out; and keep the banner off the \"Add to cart\" button. None of them changes how the page looks to a person. As we've argued before, accessibility work is agent readiness work.
\nPeak season is when a broken journey costs the most, and it's also when the riskiest changes go live: tighter bot rules, sale prices, promo pop-ups, then a code freeze that makes anything you missed hard to fix. Here's what to check before the freeze, so AI agents shopping for your customers can still find you, read your prices and buy.
\nEvery change made for peak is reasonable on its own. Security tightens bot protection to stop scalpers and card testing. Marketing adds a countdown banner and an email pop-up. Merchandising loads sale prices through a promotions app. Operations shortens cache times, or lengthens them to cope with load. Then engineering freezes the code.
\nEach of those changes is made by a different team, often in a hurry, and none of them is tested with an AI agent in mind. An assistant asked to \"find a gift under $50 that arrives before the 24th\" needs to get past your bot rules, read the sale price, see that the item is in stock, close the pop-up and reach the cart. Any one of those can fail quietly, and you won't see it in your analytics.
\n| When | What to do | Who |
|---|---|---|
| Six or more weeks out | Run AgentScore on your home page and fix anything critical. Agree which journeys matter most at peak. | E-commerce lead, developers |
| Four weeks out | Review planned bot protection changes with your CDN or security team. Check sale price and stock data on a test promotion. | Security, merchandising |
| Two weeks out | Put promo pop-ups and banners live on staging or behind a flag, and check an agent can still reach the cart. | Marketing, developers |
| Before the freeze | Rescan, set up scheduled journey tests with alerts, and agree what counts as an exception to the freeze. | E-commerce lead |
This is the change most likely to turn agents away. Common peak settings include stricter challenge thresholds, \"under attack\" modes that challenge every visitor, blocks on cloud and data center networks, and country blocks. Browser agents and assistants that fetch pages for people often run from cloud networks, so they get caught by rules written for scrapers.
\nAgentScore checks whether your bot protection treats AI assistants differently from a browser, and whether the cart and checkout show them a block or a CAPTCHA. Rescan after every bot rule change, not just once in October. For the settings themselves, see bot protection and CAPTCHAs and setting up your CDN for AI agents.
\nPromotion tools often change the price in the browser with JavaScript after the page loads. A shopper sees the sale price. An agent that reads the HTML sees the full price, or no price at all, and may quote the wrong figure to its user or skip you.
\nBefore peak, run a test promotion and check three places: the visible price in the page source (use View Source, not the inspector), the price in your product structured data, and the price in the cart. They should all agree. If a sale ends on a known date, say so:
\n\"offers\": {\n \"@type\": \"Offer\",\n \"price\": \"39.00\",\n \"priceCurrency\": \"USD\",\n \"priceValidUntil\": \"2026-12-01\",\n \"availability\": \"https://schema.org/InStock\"\n}Show the original and the sale price in text, such as \"Was $59, now $39\", rather than relying on a strikethrough alone. See prices AI agents can read.
\nAt peak, stock changes by the hour. If your structured data says InStock after an item sells out, an assistant will send people to a product they can't buy. If it says OutOfStock for an item you restocked this morning, you lose the recommendation.
availability in your product data comes from the same source as the \"Add to cart\" button, not a nightly export.PreOrder or BackOrder where they apply, and state delivery estimates in text.Email capture, spin-to-win wheels, countdown banners and free-shipping bars all multiply in November. An agent can only get past one if it has a real, labelled close button and doesn't cover the content behind it. A pop-up that sits over the \"Add to cart\" button means the agent's click lands on the pop-up instead.
\nAgentScore checks that pop-ups and banners can be dismissed, and whether the add-to-cart button on a product page is covered by something else. Check again once peak promotions are live, because they're usually added after the last scan. See cookie banners and pop-ups.
\nGift shopping by agent depends on the last few steps working. Make sure the promo code field is labelled, that a rejected code produces a clear error in text, and that the cart total updates in the page. See carts AI agents can manage and error messages AI agents can understand.
\nResist adding a CAPTCHA to checkout for peak. It stops agents at the very last step, after they've done all the work. Card testing is better handled by your payment provider's fraud tools and by rate limits on payment attempts. Keep guest checkout on. See checkout for AI shopping agents and guest checkout.
\nAssistants are often asked about delivery cut-off dates and extended returns. Put them on your shipping and returns pages as text in the HTML, with dates, not only in a banner image or a pop-up. See policy pages AI assistants can read.
\nA code freeze protects you from risky changes, but it also means a problem found in December may wait until January. Three things help:
\nBefore the freeze, check:
\nStart with a fresh AgentScore scan, then work through the list. If you have more time, our 90-day agent readiness plan covers the rest.
", "date_published": "2026-10-09T13:21:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Guide" ] }, { "id": "https://ghostagentlab.com/blog/agent-readiness-for-saas/", "url": "https://ghostagentlab.com/blog/agent-readiness-for-saas/", "title": "Agent readiness for software companies: pricing, sign-up and docs", "summary": "For software companies, the pricing page is the product page and sign-up is the checkout. How to get pricing, trials, docs, llms.txt and MCP right.", "content_html": "Most agent readiness advice is written for online stores. But software buyers send AI agents to do research too: compare plans, check whether a tool integrates with what they already use, find the limits of a free tier, and start a trial. For a software company, the pricing page is the product page, the sign-up form is the checkout, and the docs are where the real evaluation happens.
\nAgentScore looks for the pages an agent acting for a customer would need, the way an agent would: links in your home page's HTML, then your sitemap, then common addresses such as /pricing. If it finds product pages or a cart, it treats the site as a store. If it finds a pricing page and no products or cart, it treats the site as software (\"saas\" in the scan's data). That changes what some checks look for.
| Check | On a software site, it looks at |
|---|---|
| AI agents can open your product, pricing and cart pages | Whether AI assistants can load your pricing page, not just a browser |
| Prices are in the page HTML | Whether plan prices are in the HTML your server sends, before JavaScript runs |
| Key pages are linked from the home page | Whether pricing is linked from the home page with an ordinary link |
| Agents can find and press your add-to-cart or sign-up button | Whether the pricing page has a real, visible button such as \"Start free trial\" or \"Buy Pro\" |
| Cart, checkout and agentic commerce checks | Nothing. They're marked \"not tested\", so they don't count against you |
Everything else, from robots.txt and bot protection to structured data, named buttons and labelled form fields, applies the same way as for any site. See what is agent readiness? for the full model.
\nPricing pages are often the most designed page on a software site, and that's where agents struggle. Common problems:
\nA plain HTML table is still the clearest way to compare plans:
\n<table>\n <caption>Northwind plans, billed annually</caption>\n <tr><th></th><th>Starter</th><th>Pro</th></tr>\n <tr><th>Price</th><td>$12 per user per month</td><td>$29 per user per month</td></tr>\n <tr><th>Single sign-on</th><td>Not included</td><td>Included</td></tr>\n</table>For more, see prices AI agents can read.
\nAn agent that has chosen your product for someone will try to start the trial. It needs a visible, real button with a clear label, then a form it can fill in: every field labelled, errors explained in text, and no surprises. See buttons, links and forms agents can use.
\nSome steps should stay with the person, and that's fine. Email verification, single sign-on, payment details for a card-required trial and accepting terms are all reasonable places for an agent to hand back to its user. The aim is for the agent to get as far as it should, then stop cleanly with a clear message, rather than fail at a CAPTCHA on the first screen. See sign-up, booking and lead forms AI agents can finish.
\nGhost Agents can test this journey for real: from the home page to pricing to the sign-up form, using test data you supply, with a step-by-step replay showing where an agent got stuck.
\nFor many software products, the docs decide the sale. Coding assistants and research agents read them to answer \"does it support X?\" and \"how hard is it to set up?\". Treat docs with the same care as marketing pages:
\nllms.txt is a proposed convention, not a standard, and AI providers don't all say whether they read it. It's cheap to publish, though, and it suits software companies well, because your most useful pages are easy to list. Some docs platforms can generate one for you.
\n# Northwind\n> Northwind is scheduling software for field service teams.\n## Product\n- [Pricing](https://northwind.example/pricing): plans, limits and what each includes\n- [Start a free trial](https://northwind.example/signup): 14 days, no card required\n## Docs\n- [Getting started](https://northwind.example/docs/start)\n- [API reference](https://northwind.example/docs/api)\n- [Integrations](https://northwind.example/integrations)\n## Trust\n- [Security](https://northwind.example/security)\n- [Status](https://status.northwind.example/)AgentScore checks that an llms.txt file exists. It can't tell whether yours is up to date, so add it to your release checklist. See how to write an llms.txt file.
\nFor a software company, the Model Context Protocol is about more than your website. An MCP server can let an assistant use your product itself on a customer's behalf: look up records, create tasks, run reports. That's product work, with sign-in and permissions to design carefully, and many software companies are already building one.
\nThere's no single agreed way to advertise an MCP server on a website yet. AgentScore's \"Agents can use your site through MCP or WebMCP\" check looks for a server card at /.well-known/mcp.json, an MCP link in the page, a mention in llms.txt, or WebMCP tools in the page. If you have a server, mentioning it in llms.txt and your docs is the simplest start. See MCP and WebMCP for websites.
Your board doesn't need to know what robots.txt is. It needs to know what's changing, what that risks for the business, what you're doing about it, and how it will know the work is paying off. Here's a way to say all of that on one page, without hype and without numbers you can't stand behind.
\nStart with this structure. Each heading is one or two sentences. Fill in the bracketed parts with your own findings.
\nWhat's changing. People are starting to ask AI assistants to research, compare and buy on their behalf. Those assistants visit our website the way a customer would, but they read it differently.
\nWhy it matters to us. If an assistant can't get in, can't read our prices or can't complete a purchase, it recommends someone else, and we never see the lost customer in our reports.
\nWhere we stand. [Our AgentScore, the main issues it found, and whether AI agents can complete our most important journey today.]
\nWhat we're doing. [Three to five fixes, who owns each, and when they'll be done.]
\nWhat we need. [Time, tools or decisions, each with a review date.]
\nThat's it. Everything below is about filling it in well.
\nBoards have heard a lot about AI. The quickest way to lose the room is a slide of large market forecasts. You don't need them. The argument for agent readiness is modest and easy to defend:
\nIf you quote any outside figure, cite the source on the slide. If you can't, describe the trend in words instead.
\n| Risk | What it looks like | Usually owned by |
|---|---|---|
| Lost revenue | AI assistants are blocked by bot protection, can't see prices, or fail at checkout, so they recommend or buy from a competitor. | E-commerce or digital lead |
| Being misquoted | Assistants give customers the wrong price, stock or return policy because our pages are hard to read or out of date. | Marketing and content |
| Impostors | Scrapers pretend to be well-known AI agents to get past our defenses. Opening the door carelessly makes this worse. | Security |
| Content and legal | Decisions about AI training on our content, our terms of service, and responsibility for purchases an agent makes for a customer. | Legal, with marketing |
| Blind spots | Our analytics can't see most AI agents, so decisions are made on incomplete data. | Analytics or digital lead |
The point of the table is that this is a cross-team issue with no natural home. Ask the board to agree one accountable owner. See who owns agent readiness?
\nPick a few measures you can report every quarter, and show the trend rather than a single figure.
\nBe careful with attribution. You can show that more agents get through and more journeys pass. You usually can't prove exactly how much revenue that produced, and the board will trust you more if you say so. For a fuller set, see agent readiness KPIs.
\nKeep the ask small and staged, with a review point after each stage. A typical shape:
\nOur 90-day agent readiness plan maps well onto the first three stages.
\nNo. AI training crawlers and the assistants that fetch pages for a customer use different names, so you can block one and allow the other. That's a policy choice to make once, with legal. See should you block AI crawlers?
\nNot if it's done properly. The goal is to let in the agents you want, checked by verified identity, and keep blocking the rest. See how to tell if an AI crawler is real.
\nProbably nothing you'll notice, which is the problem. Some customers' assistants will fail on the site and choose a competitor, and it won't show up in any report you currently read.
\nTo fill in \"Where we stand\", run a free AgentScore scan. It takes under a minute and gives you a score out of 100 and a list of what to fix.
", "date_published": "2026-10-09T13:19:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Guide" ] }, { "id": "https://ghostagentlab.com/blog/agent-readiness-mistakes/", "url": "https://ghostagentlab.com/blog/agent-readiness-mistakes/", "title": "Seven agent readiness mistakes to avoid", "summary": "Seven common agent readiness mistakes, from blocking AI assistants with training crawlers to measuring agents with JavaScript analytics, and how to avoid each.", "content_html": "Most agent readiness problems aren't decisions anyone made. They're side effects of sensible work: a security rule, a design choice, a tracking setup. Here are seven mistakes that are easy to make and easy to miss, why each one matters, and what to do instead.
\nMany sites decide not to let AI companies train on their content, which is a fair choice. The mistake is blocking everything with \"AI\" in its name, including the assistants that fetch a page because a customer asked a question right now. Those are different agents with different names: GPTBot collects training data, while ChatGPT-User and OAI-SearchBot fetch pages for ChatGPT users and search. Anthropic draws the same line between ClaudeBot and Claude-User.
# Opt out of AI training\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: Google-Extended\nDisallow: /\n# Everyone else, including AI assistants and AI search\nUser-agent: *\nAllow: /Instead: decide on training and on assistants separately. See should you block AI crawlers? and robots.txt for AI agents. The same split applies to bot protection rules, which often block by category rather than by name.
\nA CAPTCHA at checkout is often added to stop card testing, and it works on bots. It also stops every AI agent at the very last step, after it has searched, compared, chosen a size and filled the cart. That's the most expensive place to lose a customer.
\nInstead: use your payment provider's fraud tools and rate limits on payment attempts, and keep challenges for traffic that is actually suspicious. AgentScore checks whether AI agents are blocked or shown a CAPTCHA at the cart and checkout. See checkout for AI shopping agents and bot protection and CAPTCHAs.
\nMany sites build product and pricing pages in the browser. A person sees the price a moment after the page loads. Many AI agents read the HTML your server sends and never run the scripts, so they see a product with no price, or a placeholder. They may skip you, or tell their user to check the site.
\nInstead: make sure prices are in the HTML, through server-side rendering or static generation, and in your product structured data. Check by viewing the page source, not the browser's inspector, which shows the page after scripts run. See prices AI agents can read and JavaScript-only content.
\nA magnifying glass for search, a bag for the cart, a cross to close a pop-up. People know what they mean. Browser agents choose what to click by each control's name, the same way screen readers do, and an icon with no text or label has no name at all. To the agent, the cart button isn't there.
\n<button aria-label=\"Open cart\">\n <svg aria-hidden=\"true\">…</svg>\n</button>Instead: give every icon-only button and link a label, and use real <button> and <a> elements rather than clickable boxes. See buttons, links and forms agents can use and accessibility work is agent readiness work.
llms.txt is a proposed convention for giving AI a short guide to your site. It's quick to write, which is also why it goes stale: a file written at launch still links to last year's collections, a retired plan, or a returns page that moved. An out-of-date guide can be worse than none, because it points agents at the wrong answer.
\nInstead: give llms.txt an owner and add it to the checklist for site changes, alongside the sitemap. Note that AgentScore checks that the file exists, not whether what it says is still true, so that part is up to you. See how to write an llms.txt file.
\nA user agent is a name a visitor gives itself, and anyone can use any name. If you allow a list of AI agents by user agent alone, scrapers borrow those names to get in. If you block by user agent, you may turn away a real assistant while impostors simply change their name.
\nInstead: verify identity. The large operators publish their IP ranges, and many crawlers can be confirmed with a reverse DNS lookup. Newer agents can sign their requests with Web Bot Auth, which your CDN may already check for you. See how to tell if an AI crawler is real and why we sign every request our agents send. Ghost Agent Labs' Verification page checks every request that claims to be a known agent and shows the impostors.
\nAnalytics tools such as Google Analytics count visitors by running a script in the browser. Most AI agents don't run it, so they never appear. A report that shows no AI agents isn't evidence that none came. It may mean they came and the tag didn't see them, or that they were blocked before the page loaded.
\nInstead: measure agents from your server or CDN logs, which record every request whether or not scripts run. Keep analytics for the people who click through from an assistant's answer, which it can see. See why your analytics can't see AI agents, AI referral traffic and agent traffic isn't bot traffic.
\nNone of these mistakes shows up from inside the business. The site looks fine in a browser, the dashboards look normal, and the security team sees fewer bad bots. The cost lands on customers using an assistant, who quietly go elsewhere.
\n| Mistake | Quick check | Usually fixed by |
|---|---|---|
| Assistants blocked with training crawlers | Read /robots.txt and your bot rules for AI categories | SEO, bot protection or CDN admin |
| CAPTCHA at checkout | Ask your security team what protects the checkout | Bot protection or CDN admin |
| JavaScript-only prices | View source on a product page and search for the price | Developer |
| Icons with no names | Tab through the header and listen with a screen reader | Developer |
| Stale llms.txt | Open /llms.txt and click every link | SEO or content team |
| Trusting user agents | Ask how \"allowed\" AI agents are identified | Bot protection or CDN admin |
| JavaScript analytics only | Ask where agent traffic numbers come from | Analytics or digital lead |
A free AgentScore scan catches several of these in under a minute. For the journeys themselves, Ghost Agents show whether an agent can actually finish them.
", "date_published": "2026-10-09T13:18:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Guide" ] }, { "id": "https://ghostagentlab.com/blog/how-agentscore-renders-pages/", "url": "https://ghostagentlab.com/blog/how-agentscore-renders-pages/", "title": "How AgentScore renders your pages in a real browser", "summary": "Which pages AgentScore opens in a headless browser, with what settings and safeguards, how it looks for WebMCP tools, and what it never does.", "content_html": "Some AI agents read your HTML exactly as your server sends it. Others drive a real browser, run your scripts and look at the finished page. To score your site for both, AgentScore looks at your key pages twice: once as served, and once rendered in a headless browser. Here is exactly what that browser does, how we keep it safe, and what it deliberately never does.
\nSeveral checks only make sense as a comparison between the page before and after JavaScript. Content loads without JavaScript measures how much of the rendered page's text was already in the HTML: 70% or more passes, 30% to 70% is a warning, and less than that fails. Structured data describes your business and products, Product pages give price and stock in a form agents can read and Prices are in the page HTML all look for the same thing: data that only appears once scripts have run, which agents that read the HTML never see. Our guide to JavaScript-only content covers why that matters.
\nThe Navigability checks need a rendered page too: whether a button is visible or a banner covers the screen depends on layout, which only exists once a browser has drawn the page.
\nA scan renders at most four pages, each in its own browser session:
\n| Page | When it's rendered | What we look at |
|---|---|---|
| Home page | Every scan | The Navigability checks, image alt text, text and structured data after JavaScript, WebMCP tools |
| Product page | When we find one and it loads for a browser | The add-to-cart button, prices and product data after JavaScript, WebMCP tools |
| Pricing page | When we find one and it loads for a browser | The sign-up or buy button, prices after JavaScript, WebMCP tools |
| Checkout | Only when the HTML we were sent has no form fields and the checkout didn't send us back to the cart | The form fields a checkout builds with JavaScript |
The cart is fetched but not rendered. How AgentScore works explains how key pages are found and lists every page a scan requests.
\nIf a page fails to load or render, the checks that depend on it are marked \"not tested\" with a note explaining why. A render failure never counts as a fail.
\nOn the home page, a script reads the finished page and records:
\naria-label, a referenced label, a title, or the alt text of an image inside it.<main> and <nav> landmarks, and images with no alt attribute.On product and pricing pages, we look for the main action by its text, such as \"Add to cart\", \"Buy now\", \"Start free trial\" or \"Sign up\". We scroll it into view, check whether it's a real button or link, and ask the browser what element sits at its center. If that's a cookie banner rather than the button, an agent's click would land on the banner, and Agents can find and press your add-to-cart or sign-up button fails.
\nWebMCP is a proposal for a browser feature, navigator.modelContext, that lets a page offer tools directly to an AI agent in the browser, such as \"search products\" or \"add to cart\". It isn't generally available in browsers yet, so sites that support it check for it first, roughly like this:
if (\"modelContext\" in navigator) {\n navigator.modelContext.registerTool({\n name: \"search_products\",\n description: \"Search the Northwind catalog by keyword\",\n inputSchema: { type: \"object\", properties: { query: { type: \"string\" } } },\n execute: async ({ query }) => searchCatalog(query),\n });\n}Before your scripts run, we add a stand-in for navigator.modelContext that writes down the name of every tool a page registers. A page that checks for WebMCP then registers its tools just as it would in a browser that supports it. We also look for forms marked up with a toolname attribute. We run the probe on the home, product and pricing pages and record names only: our stand-in never calls a tool. The result feeds Agents can use your site through MCP or WebMCP, where a missing setup is a warning, never a fail. Our guide to MCP and WebMCP for websites explains both.
A service that opens any URL it's given is a target: someone could submit an address that points at an internal network or a cloud metadata service. So every connection the scanner makes is checked:
\nFinishing a real task is a different job. That's what Ghost Agents do in the app: real AI agents run your journeys, stop at the payment form, and give you a step-by-step replay. The free scan marks Task completion as not tested and scores the other three categories.
\nTo reproduce a finding: open the page in a private window at desktop size and compare \"View source\" with what's on screen. If the price, product details or button is on screen but not in the source, agents that read the HTML don't get it. If a banner sits over the button on arrival, a browser agent meets it first.
\nAn AgentScore report gives you a number, a sentence and a list of checks. The number tells you where you stand. The list tells you what to do about it, as long as you read it in the right order. This guide walks through each part of the report, in the free scan and in the Ghost Agent Labs app, and shows how to turn it into a short list of fixes with an owner for each.
\nThe score is out of 100. It falls into one of four bands, and the verdict under it says what the band means:
\n| Score | Band | Verdict |
|---|---|---|
| 80 and up | Good | AI agents can use this site well. |
| 60 to 79 | Fair | AI agents can mostly use this site, but some things get in their way. |
| 40 to 59 | Weak | AI agents will struggle on this site. |
| Below 40 | Poor | Most AI agents will fail on this site. |
If any check failed, the verdict ends with Biggest issue: the summary of the failed check that carries the most weight. It's the single best place to start. If nothing failed, there's no biggest issue, and your work is in the warnings.
\nOne thing to know before you share the number: the free scan doesn't run an AI agent through a task, so Task completion shows as \"n/a\" and the score is worked out over Access, Readability and Navigability. A 72 means 72% of the points that could be tested. How AgentScore works has the arithmetic.
\nEach category gets a bar and its points, such as Access 20/25. The app shows the same thing as a share of the category's points, so Access 20/25 reads as 80/100. Use the bars to see where points are being lost, not to rank your work: a low Navigability bar with one failure in it can matter less than a single Access failure.
\nAccess comes first for a reason. If AI agents are blocked by your bot protection or robots.txt, they never see the pages the other two categories are about.
\n| Status | What it means | Counts toward the score |
|---|---|---|
| Passed | Agents will be fine here. | Full weight |
| Warning | It works, but agents will stumble, or a newer standard is missing. | Half weight |
| Failed | This stops or seriously misleads agents. | Nothing |
| Not tested | The check couldn't run on your site. | Left out entirely |
Checks are listed failures first, then warnings, then passes, then not tested. Each one has a one-line summary of what was found, often with an example, such as how many buttons have no name or which assistant was blocked.
\nA \"not tested\" check never costs you points, and its summary always says why it was skipped. The reasons fall into a few groups:
\nRead these anyway. If you run a store and the report says no product page was found from the home page or sitemap, that's worth knowing in itself: agents look for products the same way the scanner does.
\nEvery warning and failure comes with a fix: the specific change to make. In the free report, you see the findings straight away and unlock the fixes with your email address. In the app, every fix is shown.
\nThe app's Findings page adds three things that make the list easier to work through:
\nThe Readiness page shows the top four as Biggest wins. The Findings page lets you filter by severity, and Export CSV downloads every finding with its severity, score impact, summary, why it matters, fix, who usually fixes it and the scan date, ready for a ticketing tool.
\nA rule of thumb: if a finding means agents can't get in or can't see your prices, it's this sprint. If it means they'll find it a little harder, it's this quarter. If it's a new standard, it's on the roadmap. Our 90-day plan lays that out week by week.
\nA high score means your site removes the common obstacles. It doesn't prove that an AI agent can find a product, choose a size and reach the payment step. That's what Ghost Agents test in the app: real AI agents run your key journeys, stop at the payment form, and give you a step-by-step replay of where they got stuck. Use the report to clear the path, and journey tests to prove it's clear.
", "date_published": "2026-10-09T13:16:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Product" ] }, { "id": "https://ghostagentlab.com/blog/agent-traffic-is-not-bot-traffic/", "url": "https://ghostagentlab.com/blog/agent-traffic-is-not-bot-traffic/", "title": "Agent traffic isn't bot traffic", "summary": "Lumping AI agents in with scrapers hides customers. Why agent traffic needs its own categories, rules and reports, and how to start separating it.", "content_html": "Most websites sort their visitors into two piles: people and bots. People are customers. Bots are a cost, a risk, or noise to filter out of the reports. AI agents don't fit that split. Some of them are the closest thing your site has to a customer in the room, and treating them like scrapers means turning those customers away without knowing it.
\nThe people-or-bots split made sense for a long time. Apart from search engine crawlers, which everyone learned to welcome, automated traffic was mostly scrapers, credential stuffers, inventory hoarders and uptime monitors. So the tools grew up around it. Bot protection scores each request on how human it looks. Analytics filters out known bots. Dashboards show \"bot traffic\" as one line, usually as a problem.
\nAI agents arrive in that system looking like bots, because technically they are. But \"bot\" now covers software doing very different jobs for very different reasons, and some of those jobs are done for a specific person who wants to buy something.
\n| Kind | Examples | Why it's there | What it's worth to you |
|---|---|---|---|
| AI assistants | ChatGPT-User, Claude-User, Perplexity-User | A person asked a question just now, and the assistant is reading your page to answer it | A live customer question about you |
| Browser agents | Agents that run a real browser to complete a task, such as Ghost Agents | A person asked it to do something: compare, book, buy | A customer, part-way through a journey |
| Search and AI search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot | Indexing pages so they can appear in search results and AI answers later | Visibility in tomorrow's answers |
| AI training crawlers | GPTBot, ClaudeBot, CCBot, Google-Extended | Collecting content to train models | A business decision, not a customer |
| Other bots | SEO tools, uptime monitors, link previews, scrapers | Their own reasons | Mostly a cost, sometimes useful |
Google-Extended is a robots.txt token rather than a separate crawler, but it controls the same choice. Running through all five rows is a sixth problem: impostors. Scrapers often claim to be Googlebot or GPTBot to get past bot protection, so a name in a user agent proves nothing on its own.
\nThe top two rows are the ones the two-pile habit hurts most. An assistant fetch is one person's question. If it's blocked, that person gets an answer about somebody else. A browser agent that hits a CAPTCHA at checkout is an abandoned cart with no record of why. Neither shows up in your analytics, because most agents never run the tracking script, and both look like \"bots\" to a system tuned to stop bots.
\nThe fix isn't to welcome every bot. It's to stop treating them as one thing. In practice that means four habits:
\nOne honest caveat: not every agent can be identified. Some browser agents use an ordinary browser's user agent and run from cloud addresses, so they look like people, or like a bot your protection doesn't recognize. Signed requests will help as more operators adopt them. Until then, the best evidence of how these agents fare is to run AI agents through your journeys yourself.
\nThe most useful change is often the simplest: stop putting AI agents in the \"bots\" line. Give assistants, browser agents, AI search and training crawlers their own lines, alongside people, and report what each one got back. When a leader asks \"is AI a threat or an opportunity for us?\", that report answers with your own numbers instead of an opinion. Our guide to agent readiness KPIs suggests what to put in front of them each month, and tracking AI referral traffic covers the visits and sales that follow.
\nIn the Ghost Agent Labs app, Agent traffic does this split from your server or CDN logs: people, search crawlers, AI training crawlers, AI assistants and autonomous agents, with each assistant's fetches grouped into sessions and a breakdown of how often agents were served, redirected, blocked or sent to missing pages. Verification separates verified agents from impostors.
\nYour next customer may arrive as a request with a user agent you've never looked at. It's worth knowing which pile you've put it in.
", "date_published": "2026-10-09T13:15:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Perspective" ] }, { "id": "https://ghostagentlab.com/articles/cdn-settings-ai-agents/", "url": "https://ghostagentlab.com/articles/cdn-settings-ai-agents/", "title": "Setting up your CDN for AI agents", "summary": "How to set up your CDN and WAF for AI agents: verified bot categories, signed requests, rule order, challenge versus block, and logging.", "content_html": "Your CDN and web application firewall (WAF) decide, request by request, who gets to see your site. Most were tuned before AI agents were a real source of customers. This guide walks through the settings that matter, in the order your CDN applies them, so verified AI agents get your pages and the bots you don't want still get stopped.
\nIf you're still working out why agents get caught in the first place, start with Bot protection and CAPTCHAs: stop blocking the agents you want. This article is the practical follow-on: how to set the CDN up, and how to check it stays that way.
\nBefore anyone changes a rule, agree which kinds of agent you want. Most businesses land somewhere like this:
\n| Kind of agent | What it does | Typical choice |
|---|---|---|
| AI assistants (for example ChatGPT-User, Claude-User, Perplexity-User) | Fetch a page because a person asked a question about it right now | Allow |
| AI search crawlers (for example OAI-SearchBot) | Index pages so they can appear in AI search answers | Allow |
| Search engines (Googlebot, Bingbot) | Classic search indexing, which also feeds some AI features | Allow |
| AI training crawlers (for example GPTBot, ClaudeBot) | Collect content to train models | A business choice; blocking is common |
| Unverified bots claiming any of the names above | Unknown | Challenge, rate limit or block |
Write the decision down. It's what your CDN admin implements, and it's what you check against later. Our guides to robots.txt for AI agents and whether to block or allow AI crawlers cover the trade-offs.
\nMost large CDNs and bot management products keep a list of verified bots: crawlers and agents whose requests they have confirmed really come from the operator they name. Many also group those bots into categories, so you can write one rule for \"AI assistants\" instead of maintaining a list of names yourself.
\nBuild your rules on that verification, not on the User-Agent header. Anyone can send a request that says it's ChatGPT-User. A rule that skips bot checks for that string is an open door for scrapers. Our guide to telling real AI crawlers from fakes explains the methods CDNs use behind the scenes: published IP ranges, reverse DNS and signatures.
Check the category names in your own console. Vendors name and split these categories differently, and they change them as new agents appear. Confirm which category each agent you care about sits in before you write a rule that depends on it.
\nIP-based verification struggles with agents that run in the cloud, because their addresses change and are often shared. Web Bot Auth fixes that by having the agent sign each request with a private key, and publish the matching public key on its own domain. It builds on HTTP Message Signatures (RFC 9421), and is still being standardized, so support varies.
\nIf your CDN can verify these signatures, turn it on and treat a valid signature as verification. If it can't yet, nothing breaks: signed requests are ordinary requests with extra headers. Our post Why we sign every request our agents send explains how it works in practice.
\nCDNs apply rules in a sequence, and the first matching action often wins. Most agent problems we see come from order, not from a missing rule. A sensible order looks like this:
\nThe common mistake is a broad \"block AI bots\" or \"challenge automated traffic\" rule placed above the verified-agent allow rule. The allow rule then never runs.
\nMost bot tools offer several actions. For a person they feel very different. For an AI agent, most of them end the same way.
\n| Action | What a person sees | What an AI agent gets |
|---|---|---|
| Block | An error page | An error page; it tells the user it can't reach you |
| JavaScript or \"managed\" challenge | A short wait, usually invisible | Usually a dead end: most assistant fetchers don't run scripts |
| Interactive challenge or CAPTCHA | A puzzle or checkbox | A dead end: agents can't, and shouldn't, solve them |
| Rate limit (429 with Retry-After) | Rarely seen | A clear signal to slow down and try again |
| Log only | Nothing | The page |
So treat a challenge as a block when you're thinking about agents. Keep challenges for traffic you genuinely don't want, and for high-risk actions such as login, account creation and payment. Product, pricing, policy and help pages should open for verified agents without one.
\nWhen an assistant says it \"couldn't access\" your site, someone has to find out why. Make sure your CDN logs, or the samples it keeps, include for each request:
\nBefore you switch a new rule to block, run it in log-only (sometimes called \"simulate\" or \"count\") mode for a few days and check what it would have caught. Then review blocked and challenged agent traffic regularly. Our guides to measuring AI agent traffic and finding the errors AI agents hit in your logs show what to look for.
\nProducts change quickly and menus move, so treat these as pointers, and check your vendor's current documentation for the exact names and the plan you're on.
\nIf you use a dedicated bot product in front of or behind your CDN, the same questions apply to it. Two layers mean two places an agent can be stopped.
\nA configuration that looks right in the console can still behave differently in practice. Test it as an agent would:
\nOne caution: AgentScore's requests under those agent names aren't signed and don't come from those operators' networks, so a correctly configured CDN that only allows verified agents may still challenge them. That's the right behavior. Read a failure alongside your logs: if real, verified agent traffic is getting through, your setup is working. Our post on why our scanner wears other names explains the trade-off.
", "date_published": "2026-10-09T13:14:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Access" ] }, { "id": "https://ghostagentlab.com/articles/geo-blocking-ai-agents/", "url": "https://ghostagentlab.com/articles/geo-blocking-ai-agents/", "title": "Geo-blocking, VPN blocks and AI agents", "summary": "Why country, VPN and data center blocks and IP-based redirects catch AI agents visiting from cloud servers abroad, and how to let verified agents through.", "content_html": "A shopper in Manchester asks an AI assistant about your store. The assistant doesn't visit from Manchester. It visits from a cloud data center, quite possibly in another country. If your site blocks, challenges or redirects visitors by location or network, that one fact can decide whether the assistant sees your real store, the wrong store, or nothing at all.
\nWhen a person asks an assistant a question, the page request doesn't come from their phone or laptop. It comes from the assistant's servers. The same is true for AI search crawlers and for browser agents that run in the cloud. That has three consequences:
\nNone of this is a sign of bad intent. It's simply how AI agents work. But it means rules written with human visitors in mind can treat agents very differently from the people they're acting for.
\nSome sites block whole countries, often to cut fraud or because they don't ship there. If an assistant's servers sit in a blocked country, every request it makes for every shopper is turned away, including shoppers in countries you serve.
\nBlocking \"anonymizing\" networks or hosting providers is a common anti-fraud and anti-scraping setting. It catches nearly all AI agent traffic, because nearly all of it comes from hosting providers. It's one of the most common reasons an assistant can reach a competitor's site but not yours.
\nMany international stores redirect visitors to a country site based on IP address. An agent fetching your UK product page for a UK shopper may be sent to your US site instead, then quote US prices, US stock and US delivery times. The answer looks confident and is wrong. Some redirects go to a country picker page, which leaves the agent with no product at all.
\nSome sites quietly change prices, currency, product ranges or policies based on the visitor's location, without changing the URL. An agent then describes a version of your store the shopper will never see.
\nThe aim is to keep the protection you need while letting verified agents see the right content.
\nSome location rules are required: sanctions, licensing, or content you're not allowed to show in certain places. Keep those, and get advice on how they apply to automated visitors. Many others are preferences, such as \"we don't ship there, so block it\". For those, ask whether blocking the server's location actually achieves anything. You already check the delivery address at checkout, which is where a shipping restriction belongs.
\nMost CDNs and bot management tools can tell verified bots apart from other traffic. Add an exception so that verified AI assistants and AI search agents skip the country, VPN and data center rules that are about fraud or scraping. Put it in the right place in your rule order, and never base it on the user agent alone. Our guide to setting up your CDN for AI agents covers rule order, and telling real AI crawlers from fakes covers verification.
\nDon't allow whole cloud providers. Lifting a data center block for an entire cloud network lets in every scraper hosted there. Allow verified agents, and keep the block for everything else.
\nGive each country or language its own URL, such as northwind.example/en-gb/ or uk.northwind.example, and serve the content that URL promises, whoever asks. If you want to steer people to their local site, show a banner suggesting it instead of redirecting. Agents and people can then follow the link that matches the shopper. Mark the alternatives with hreflang so agents and search engines can find them:
<link rel=\"alternate\" hreflang=\"en-gb\" href=\"https://northwind.example/en-gb/boots/trail-runner/\">\n<link rel=\"alternate\" hreflang=\"en-us\" href=\"https://northwind.example/en-us/boots/trail-runner/\">\n<link rel=\"alternate\" hreflang=\"x-default\" href=\"https://northwind.example/boots/trail-runner/\">Our guide to languages, currencies and regions goes further on international setups.
\nSome agents will still land on your default site. Make sure it's a real store, not a country picker. State which country and currency the prices are in, and link clearly to the other country sites. Put the same information in your structured data, so an agent knows the price it read is in US dollars and not pounds. See machine-readable prices.
\nFor traffic you can't verify, from data centers or anywhere else, rate limits are usually a better tool than outright blocks. They stop bulk scraping while still letting through a small number of requests made on someone's behalf. See rate limits for AI agents.
\n| Look for | Where | What it suggests |
|---|---|---|
| Country, ASN, \"hosting\" or \"anonymous proxy\" rules | CDN, WAF or bot management settings | Agents may be blocked or challenged by location or network |
| Redirects based on IP country | CDN rules, e-commerce platform settings, geolocation apps or plugins | Agents may see the wrong country's store |
| 403s, challenges or 302 redirects for ChatGPT-User, Claude-User or Perplexity-User | CDN or server logs | Real agent visits are being turned away or sent elsewhere |
| Assistants quoting the wrong currency or prices | Ask a few assistants about your products | A redirect or location-based content is reaching agents |
An AgentScore scan helps too. Its requests come from cloud servers, like real agents' requests do, and it compares what AI agents get from your home page, product, pricing and cart pages with what a normal browser gets from the same servers. It can't show what visitors in every country see, so a block that only applies to certain countries may not show up. Pair it with a look at your rules and logs. Our guide to finding the errors AI agents hit in your logs shows how to pull out agent requests and their status codes.
\nhreflang and a suggestion banner.An AI agent acting for a customer usually doesn't have that customer's password, and it can't create accounts or pay for subscriptions on a whim. Anything you put behind a login or a paywall is, for most agents, simply not there. That's often the right call. But many sites hide things that should be public, and lose recommendations because of it.
\nDifferent kinds of agent meet a login wall in different ways:
\n| Kind of agent | Behind a login or paywall |
|---|---|
| AI assistants fetching a page for a question | Get the login page or the paywall, and answer from whatever is visible |
| AI search and training crawlers | Index only what's public, unless you choose to show them more |
| Browser agents carrying out a task | Usually stop and ask the person to sign in, or give up. Some can hand control back to the person for the sign-in step |
So the question for each page is simple: if an agent can only see the public version, can it still answer the customer's question and move them forward?
\nThese are the things agents are most often asked about. If they sit behind a login, the agent either says it doesn't know or quotes a competitor who shows them openly.
\nTrade and wholesale businesses often hide everything, because prices vary by customer. Consider showing the catalog and list prices publicly, with a note that account holders get their own prices after signing in. The agent can then recommend you and send the buyer to log in for the final number.
\nPlenty of content belongs there: order history, saved addresses, account settings, customer-specific prices, members-only content and paid articles. The goal isn't to remove logins. It's to make sure the wall sits in front of private and paid things, not in front of the information a customer needs to decide.
\nWhen an agent does reach something that needs the customer, make the handoff smooth:
\nautocomplete values help browser agents and password managers alike. See sign-up, booking and lead forms AI agents can finish.Publishers face a specific version of this. You may want search engines to see the full article so they can rank it, while readers see a paywall. Showing crawlers different content from readers can look like cloaking. Google's documented answer is paywalled content structured data: you mark the article as not free, and point to the part that's behind the paywall.
\n<script type=\"application/ld+json\">\n{\n \"@context\": \"https://schema.org\",\n \"@type\": \"NewsArticle\",\n \"headline\": \"How trail running shoes are tested\",\n \"isAccessibleForFree\": false,\n \"hasPart\": {\n \"@type\": \"WebPageElement\",\n \"isAccessibleForFree\": false,\n \"cssSelector\": \".paywalled-content\"\n }\n}\n</script>The cssSelector must match a class on the element that wraps the paid content. Google explains the details in its paywalled content guidelines, and the properties are defined at schema.org.
Two things to keep in mind:
\nBefore an AI agent recommends a store, it wants to know who it's dealing with: the business name, how to contact it, where it ships from, when it's open, and what happens if something goes back. Organization data puts those facts in one place, in a form an agent can read without piecing them together from your footer.
\nPeople ask AI assistants questions like \"is this a real shop?\", \"do they have a phone number?\", \"are they open on Sunday?\" and \"can I return it if it doesn't fit?\". To answer, an agent has to find those facts somewhere on your site. Usually they're scattered: the phone number in the footer, the address on a contact page, the returns window in a policy page, the opening hours in an image.
\nAn agent can often work it out, but every guess is a chance to get it wrong or to give up and recommend someone whose details are easier to confirm. Structured data about your business gives it one trusted summary, on the page it's most likely to visit first: your home page.
\nProduct markup describes what you sell. Organization markup describes who is selling it. Agents use both. For products, see product data that AI shopping agents can read.
\nSchema.org has a family of types for businesses. Use the most specific one that's true.
\n| Type | Use it when |
|---|---|
Organization | The general case: any company, brand or nonprofit |
OnlineStore | You sell online and have no shop customers can walk into. It's a more specific kind of Organization |
LocalBusiness or a subtype such as Store | Customers can visit you at a physical address, such as a shop, showroom or clinic |
If you have an online store and physical shops, describe the company as an Organization or OnlineStore on the home page, and give each shop its own LocalBusiness (or Store) markup on its own location page, with its own address and hours.
Here's JSON-LD for the home page of a fictional online coffee store. It goes in the page's HTML, inside the <head> or <body>:
<script type=\"application/ld+json\">\n{\n \"@context\": \"https://schema.org\",\n \"@type\": \"OnlineStore\",\n \"@id\": \"https://northwind.example/#organization\",\n \"name\": \"Northwind Coffee\",\n \"legalName\": \"Northwind Coffee Ltd\",\n \"url\": \"https://northwind.example/\",\n \"logo\": \"https://northwind.example/images/logo.png\",\n \"description\": \"Specialty coffee roasted to order and shipped across the US.\",\n \"email\": \"help@northwind.example\",\n \"telephone\": \"+1-555-010-0199\",\n \"address\": {\n \"@type\": \"PostalAddress\",\n \"streetAddress\": \"120 Harbor Street\",\n \"addressLocality\": \"Portland\",\n \"addressRegion\": \"OR\",\n \"postalCode\": \"97201\",\n \"addressCountry\": \"US\"\n },\n \"contactPoint\": [{\n \"@type\": \"ContactPoint\",\n \"contactType\": \"customer service\",\n \"email\": \"help@northwind.example\",\n \"telephone\": \"+1-555-010-0199\",\n \"availableLanguage\": [\"en\"],\n \"hoursAvailable\": {\n \"@type\": \"OpeningHoursSpecification\",\n \"dayOfWeek\": [\"Monday\", \"Tuesday\", \"Wednesday\", \"Thursday\", \"Friday\"],\n \"opens\": \"09:00\",\n \"closes\": \"17:00\"\n }\n }],\n \"sameAs\": [\n \"https://www.instagram.com/northwindcoffee.example\",\n \"https://www.linkedin.com/company/northwind-coffee-example\"\n ],\n \"hasMerchantReturnPolicy\": {\n \"@type\": \"MerchantReturnPolicy\",\n \"applicableCountry\": \"US\",\n \"returnPolicyCategory\": \"https://schema.org/MerchantReturnFiniteReturnWindow\",\n \"merchantReturnDays\": 30,\n \"returnMethod\": \"https://schema.org/ReturnByMail\",\n \"returnFees\": \"https://schema.org/FreeReturn\",\n \"merchantReturnLink\": \"https://northwind.example/returns\"\n }\n}\n</script>name, and your registered company name in legalName.ContactPoint for each way to reach you, with contactType saying what it's for (\"customer service\", \"sales\", \"technical support\"). Add the hours it's staffed and the languages it covers, so an agent can tell someone whether they'll get an answer today.PostalAddress answers \"where are they based?\" and helps an agent judge shipping times and which country's consumer rules apply. Only include it if you're happy for it to be quoted.sameAs. Links to your official profiles elsewhere: social accounts, marketplace storefronts, a Wikipedia or business-register entry. They help an agent connect your site to what others say about you, and make it harder to confuse you with a similarly named business.hasMerchantReturnPolicy on the organization sets a default for everything you sell. Google added support for return policies at the organization level, so you don't have to repeat it on every product. Product-level policies can still override it where they differ.For a shop, showroom or other location, the most common question is \"are they open now?\". Use openingHoursSpecification on the LocalBusiness, one entry per set of days with the same hours:
\"openingHoursSpecification\": [\n { \"@type\": \"OpeningHoursSpecification\",\n \"dayOfWeek\": [\"Monday\", \"Tuesday\", \"Wednesday\", \"Thursday\", \"Friday\"],\n \"opens\": \"08:00\", \"closes\": \"18:00\" },\n { \"@type\": \"OpeningHoursSpecification\",\n \"dayOfWeek\": \"Saturday\", \"opens\": \"09:00\", \"closes\": \"16:00\" }\n]Leave out days you're closed. For holidays, add an entry with validFrom and validThrough dates, so an agent doesn't send someone to a locked door on December 25. Add geo coordinates and a telephone for each location too.
@id (like https://northwind.example/#organization) so other markup on your site, such as a product's brand or a WebSite's publisher, can point to the same organization instead of repeating it.The AgentScore check \"Structured data describes your business and products\" reads your home page's HTML for schema.org JSON-LD:
\nOrganization and WebSite.The check confirms the markup is there and readable. It doesn't judge whether your hours or phone number are right, so validate the content yourself.
\napplication/ld+json. Many platforms and themes already output a basic Organization block with just a name and logo.Business details are one part of being readable to AI agents. For the rest, see the agent readiness guide, and for the details shoppers ask about most, reviews and ratings AI agents can read.
", "date_published": "2026-10-09T13:11:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/product-descriptions-ai-agents/", "url": "https://ghostagentlab.com/articles/product-descriptions-ai-agents/", "title": "Product descriptions AI agents can compare", "summary": "How to write product descriptions AI agents can compare: specs as text and tables, consistent attributes, units, materials and compatibility.", "content_html": "When someone asks an AI agent for \"a waterproof hiking jacket under 500 grams that packs into its own pocket\", the agent has to read your product pages and compare them with everyone else's. It can only recommend what it can confirm. Product descriptions written as facts, not mood, are what get you onto the shortlist.
\nA person browsing can look at a photo, skim the copy and get a feel for a product. An AI agent works differently. It turns the shopper's request into requirements (waterproof, under 500 grams, packable, under $200) and then checks each product page for evidence that the product meets them.
\nIf your page says \"featherlight protection for every adventure\", the agent can't match that to \"under 500 grams\". If a competitor's page says \"Weight: 410 g (size M)\", it can. The agent recommends the product it can back up, and yours drops out, even if it's the better jacket.
\nThis doesn't mean brand voice has to go. It means the facts need to be on the page too, written so they can be read and compared on their own.
\nAgents read text. Specifications that live only in an image, a size-chart graphic or a downloadable PDF are often invisible to them. The same goes for details that only appear when someone clicks a tab or accordion, if that content is loaded by JavaScript rather than present in the HTML. (Content hidden by CSS until clicked is fine; it's still in the page.)
\nPut the key facts in the page's HTML as plain text. A simple table works well, because each row is a clear pair of attribute and value:
\n<table>\n <caption>Specifications</caption>\n <tr><th scope=\"row\">Weight</th><td>410 g (size M)</td></tr>\n <tr><th scope=\"row\">Waterproof rating</th><td>20,000 mm hydrostatic head</td></tr>\n <tr><th scope=\"row\">Shell fabric</th><td>100% recycled nylon, 3-layer</td></tr>\n <tr><th scope=\"row\">Packed size</th><td>18 x 12 x 7 cm, packs into chest pocket</td></tr>\n <tr><th scope=\"row\">Fit</th><td>Regular, room for a fleece underneath</td></tr>\n</table>A definition list (<dl>) works just as well. What matters is that each fact has a label next to it, in text.
Comparison only works when the same attribute is described the same way across your catalog. Agree on a set of attributes for each category and fill them for every product.
\n| Do | Avoid |
|---|---|
| Weight: 410 g | Ultralight |
| Battery life: up to 14 hours of video playback | All-day battery |
| Capacity: 1.7 liters (7 cups) | Family size |
| Dimensions: 60 x 40 x 25 cm (W x D x H) | Compact design |
| Material: 100% organic cotton, 180 gsm | Premium soft fabric |
| Fits: iPhone 15 and iPhone 16 (not Pro Max) | Fits most phones |
Use the same attribute names, the same order and the same units on every product in a category. An agent comparing your three kettles should find \"Capacity\" in the same place on each page.
\nFor parts, accessories, consumables and refills, compatibility is the whole decision. A shopper asks \"will this filter fit my Model X200?\" and the agent needs a clear yes or no.
\nIn structured data, schema.org has properties such as isAccessoryOrSparePartFor and isConsumableFor that point from your product to the product it's used with.
Product JSON-LD is where agents look first for price and stock. It can carry descriptive attributes too, so they don't have to be read out of your layout:
\n{\n \"@context\": \"https://schema.org\",\n \"@type\": \"Product\",\n \"name\": \"Ridgeline Packable Rain Jacket, Men's\",\n \"sku\": \"NW-RJ-M-BLU\",\n \"material\": \"100% recycled nylon\",\n \"color\": \"Blue\",\n \"size\": \"M\",\n \"weight\": { \"@type\": \"QuantitativeValue\", \"value\": 410, \"unitCode\": \"GRM\" },\n \"additionalProperty\": [\n { \"@type\": \"PropertyValue\", \"name\": \"Waterproof rating\",\n \"value\": 20000, \"unitText\": \"mm\" },\n { \"@type\": \"PropertyValue\", \"name\": \"Packable\", \"value\": true }\n ],\n \"offers\": {\n \"@type\": \"Offer\",\n \"price\": \"189.00\",\n \"priceCurrency\": \"USD\",\n \"availability\": \"https://schema.org/InStock\"\n }\n}Use additionalProperty for attributes schema.org has no dedicated property for. The values must match what the page says. For the full product markup, including variants, returns and shipping, see product data that AI shopping agents can read.
Marketing copy still has a job: it persuades the person who reads it. The problem is when it replaces facts instead of sitting beside them. Watch for these:
\nA good pattern: one or two sentences of plain summary (what it is, who it's for, the one or two facts that matter most), then your brand copy, then the full specification table.
\nAgentScore doesn't grade the wording of your descriptions. On a product page, it checks that schema.org Product data gives price, currency and availability (\"Product pages give price and stock in a form agents can read\"), and that prices are in the page HTML. To see how a real agent handles your descriptions, run a journey with Ghost Agent, such as \"find a waterproof jacket under 500 grams\", and read the step-by-step replay. See testing journeys with AI agents.
\nShoppers also lean on prices, variants and reviews when comparing; see machine-readable prices, variant pickers AI agents can use and reviews and ratings AI agents can read.
", "date_published": "2026-10-09T13:10:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/reviews-ratings-ai-agents/", "url": "https://ghostagentlab.com/articles/reviews-ratings-ai-agents/", "title": "Reviews and ratings AI agents can read", "summary": "How to make reviews and ratings visible to AI agents: reviews in the HTML, AggregateRating and Review markup that follows the rules, and honest moderation.", "content_html": "\"Which of these has the best reviews?\" is one of the most natural questions to ask an AI shopping assistant. To answer it, the agent has to find your reviews, read them and trust them. Many stores load reviews in a way agents never see, or mark them up in ways that break the rules. Here's how to get it right.
\nAI agents use reviews in two ways. The rating is a quick signal when they rank options: a 4.7 from several hundred reviews reads differently from a 4.7 from three. The review text answers specific questions the product description doesn't: \"does it run small?\", \"is it loud?\", \"how long did the battery last after a year?\".
\nIf an agent can't see your reviews, it has nothing to weigh. It may fall back on reviews it finds elsewhere, or simply prefer a competitor whose ratings it can confirm.
\nMost stores use a review app or service. Many of these load reviews with JavaScript after the page opens, from a separate server. A person sees them a moment later. An AI agent that reads the HTML without running JavaScript sees an empty space where the reviews should be, and many agents work that way.
\nFor the wider problem of content that only appears after JavaScript runs, see why AI agents can't see JavaScript-only content.
\nSchema.org gives agents the same facts in a standard form. On a product page, add aggregateRating for the summary and review for individual reviews, inside your Product JSON-LD:
{\n \"@context\": \"https://schema.org\",\n \"@type\": \"Product\",\n \"name\": \"Ethiopia Yirgacheffe Whole Bean Coffee, 340 g\",\n \"aggregateRating\": {\n \"@type\": \"AggregateRating\",\n \"ratingValue\": \"4.6\",\n \"bestRating\": \"5\",\n \"reviewCount\": \"318\"\n },\n \"review\": [{\n \"@type\": \"Review\",\n \"author\": { \"@type\": \"Person\", \"name\": \"Dana R.\" },\n \"datePublished\": \"2026-09-14\",\n \"reviewRating\": { \"@type\": \"Rating\", \"ratingValue\": \"5\", \"bestRating\": \"5\" },\n \"reviewBody\": \"Bright and floral as a pour-over. Arrived two days after roasting.\"\n }],\n \"offers\": {\n \"@type\": \"Offer\",\n \"price\": \"18.00\",\n \"priceCurrency\": \"USD\",\n \"availability\": \"https://schema.org/InStock\"\n }\n}Use reviewCount for reviews with text, or ratingCount if you count star-only ratings too. Say what scale you use with bestRating if it isn't 1 to 5.
Search engines publish rules for review markup, and following them is the safest guide for AI agents too. The key ones from Google's review snippet guidelines:
\nLocalBusiness and Organization pages. The safe reading is: put ratings on your products, and leave your company-level reputation to independent review sites.Google's rules apply to its search results, not to every AI agent. But they describe what makes review data trustworthy, and agents that cross-check your markup against the page will notice the same problems.
\nAI agents read across many sources. If your site shows a perfect 5.0 and independent review sites say 3.2, the gap is itself a signal, and an agent may say so to the shopper. The only durable approach is reviews that reflect what customers actually think.
\nAgentScore doesn't have a dedicated reviews check. Its structured data checks look for schema.org JSON-LD on your home page, and for Product data with price, currency and availability on a product page, and flag markup that only appears after JavaScript runs. If your review app adds its markup with JavaScript, the same problem applies to your ratings. To see what an agent actually makes of your reviews, run a Ghost Agent journey such as \"find the best-rated coffee under $20 and add it to the cart\" and read the replay.
\naggregateRating that matches the page.Reviews work best alongside clear facts. See product descriptions AI agents can compare and product data that AI shopping agents can read.
", "date_published": "2026-10-09T13:09:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/canonical-urls-ai-agents/", "url": "https://ghostagentlab.com/articles/canonical-urls-ai-agents/", "title": "Canonical URLs and duplicate pages for AI agents", "summary": "How to stop AI agents reading the same page at many URLs: rel=canonical, parameter and variant URLs, http, https and www redirects, and clean sitemaps.", "content_html": "Most online stores show the same product at many different addresses: with tracking tags, with a color selected, under two categories, on http and https. People don't notice. AI agents do, because each address can look like a different page. Canonical URLs tell them which one is the real one.
\nWhen an AI agent reads your site, or when an AI search service builds the index an assistant draws on, it meets your pages by URL. If the same jacket appears at five URLs, it may:
\nThe fix is not to make duplicates disappear, which is often impossible, but to say clearly which URL is the main one, and to make every other signal agree.
\n| Source | Example |
|---|---|
| Tracking parameters | /products/rain-jacket?utm_source=newsletter |
| Sort and filter parameters | /jackets?sort=price&color=blue |
| Variant parameters | /products/rain-jacket?variant=4417 |
| Category paths | /mens/jackets/rain-jacket and /sale/rain-jacket |
| Protocol and host | http:// and https://, with and without www. |
| Trailing slashes and case | /Rain-Jacket/ and /rain-jacket |
A canonical link is one line in the page's <head> that names the preferred URL for the content:
<link rel=\"canonical\" href=\"https://northwind.example/products/rain-jacket\">Put it on every indexable page, including on the canonical page itself (a self-referencing canonical). Rules that keep it reliable:
\nhttps:// and the host, not just the path.noindex.Search engines treat rel=\"canonical\" as a strong hint rather than a command, and the same is likely true of AI services that build on them. That's why the other signals below need to agree with it.
Tracking parameters like utm_source and click IDs never change the content, so the page should always name the clean URL as canonical. Sort orders and session IDs are the same.
Filters are a judgment call. A filtered listing such as \"blue jackets\" may be worth keeping as its own page if people search for it. If so, give it a clean path (/jackets/blue), its own title and a self-referencing canonical. Most combinations of filters, though, should point back to the unfiltered category. For more on filters agents can use, see site search and filters for AI agents.
Variants are where stores most often go wrong. There are two reasonable patterns:
\n?variant=4417, ?color=blue) has a canonical pointing to the main product URL. This suits products where variants differ only in size or color and share a description. Make sure the main page lists every variant's price and availability, ideally in ProductGroup markup, so the agent doesn't lose them.What doesn't work is a mix: variant URLs that canonicalize to the main page, while the main page only shows the default variant's price and stock. An agent is then told the blue jacket is the same page as the red one, but can't find the blue one's details. See size, color and variant pickers AI agents can use.
\nFor duplicates that are purely technical, a canonical tag isn't enough. Send a permanent (301 or 308) redirect so that every visitor, person or agent, lands on one version:
\nhttp:// redirects to https://.www.northwind.example or northwind.example, and redirect the other.Redirect in a single hop. Chains (http to https, then to www, then to a trailing slash) slow every request down, and some clients stop following after a few hops. Avoid redirects that depend on JavaScript or a meta refresh; agents that don't run scripts get stuck on the first page. And make sure region or language redirects don't trap agents, as covered in international sites for AI agents.
\nAgents and crawlers look at more than the canonical tag. Each of these should use the same, canonical URL:
\nurl in your Product and Offer JSON-LD should match the canonical.og:url and feeds. Social tags and product feeds should use the same address.AgentScore doesn't have a dedicated canonical check. The \"Valid sitemap\" check confirms you have a sitemap it can read (found through robots.txt, /sitemap.xml or /sitemap_index.xml) and counts its entries; it doesn't judge whether each URL is canonical. AgentScore also uses your sitemap and home page links to find a product page to test, so a sitemap full of redirecting or parameter URLs makes that harder for any agent.
?utm_source=test on the end. View the source and check the canonical points to the clean URL.http:// and the non-preferred host into a browser. Do they each redirect, in one hop, to the right page?Canonical URLs are housekeeping, but they decide which version of your page agents read and repeat. For the bigger picture, see the agent readiness guide.
", "date_published": "2026-10-09T13:08:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/international-sites-ai-agents/", "url": "https://ghostagentlab.com/articles/international-sites-ai-agents/", "title": "Languages, currencies and regions: international sites for AI agents", "summary": "How to make international stores work for AI agents: lang and hreflang, unambiguous currencies, region redirects that don't trap agents, and shipping countries.", "content_html": "If you sell in more than one country, an AI agent needs to know which version of your site it's reading: what language it's in, which currency the prices are in, and whether you'll deliver to the shopper's address. Get this wrong and an agent may quote a euro price to someone in Ohio, or never get past your country picker at all.
\nInternational sites are built for people, who can see a flag in the corner, read a price symbol and click \"change country\". AI agents work from the page's code and text, often from a server in another country, and usually without the browser settings that a person's visit carries. Common failures:
\nEvery page should say what language it's in, on its <html> tag:
<html lang=\"en-GB\">Use a language code, optionally with a region: en, en-GB, fr-CA, de-CH. It tells agents, screen readers and translation tools how to read the page. Make sure it changes with the language version. A French page that still says lang=\"en\", because the theme hard-codes it, is surprisingly common. The AgentScore check \"Clear page title, description, and headings\" flags a missing lang attribute on your home page; see titles, descriptions and headings.
hreflang links tell agents and search engines that several pages are versions of the same content for different languages or regions, and which one is for whom. Each version lists all the others, and itself:
<link rel=\"alternate\" hreflang=\"en-us\" href=\"https://northwind.example/us/products/rain-jacket\">\n<link rel=\"alternate\" hreflang=\"en-gb\" href=\"https://northwind.example/uk/products/rain-jacket\">\n<link rel=\"alternate\" hreflang=\"fr-fr\" href=\"https://northwind.example/fr/produits/veste-pluie\">\n<link rel=\"alternate\" hreflang=\"x-default\" href=\"https://northwind.example/products/rain-jacket\">href should be the canonical URL of that version, returning 200, not a redirect. See canonical URLs and duplicate pages.x-default. It names the page for everyone else, typically a global version or a country picker that works without redirects.en-gb, not uk or gb-en.You can put hreflang in the page <head>, in HTTP headers, or in your XML sitemap. Pick one method and keep it consistent.
Many stores redirect visitors automatically based on their IP address or browser language. For people, it saves a click. For AI agents it can be a dead end. Agents often run from cloud data centers, so the location you detect is the server's, not the shopper's. An agent helping someone in Berlin may be sent to your US store, and if every attempt to open /de/ bounces back to /us/, it can never read the German prices.
/de/products/rain-jacket, serve it.A symbol alone is often not enough. \"$\" is used by more than a dozen currencies, and \"kr\" by several. Show the currency code where there's any doubt (US$189 or 189.00 USD), and always put it in your structured data:
\"offers\": {\n \"@type\": \"Offer\",\n \"price\": \"175.00\",\n \"priceCurrency\": \"GBP\",\n \"availability\": \"https://schema.org/InStock\"\n}priceCurrency takes a three-letter ISO 4217 code: USD, GBP, EUR, CAD. Each country version should show its own price and currency in both the page and the markup. Say whether prices include sales tax or VAT, since that differs by country. More in machine-readable prices.
Watch out for converted prices. If prices are converted on the fly with JavaScript, an agent that doesn't run JavaScript sees your base currency, while the shopper would have paid something else. Render each store's price in the HTML.
\n\"Do they deliver to Canada?\" is a question an agent should be able to answer before sending someone to your checkout. Make it easy:
\nshippingDetails on an Offer can describe where you deliver, at what cost, and how long it takes:\"shippingDetails\": {\n \"@type\": \"OfferShippingDetails\",\n \"shippingDestination\": { \"@type\": \"DefinedRegion\", \"addressCountry\": [\"GB\", \"IE\"] },\n \"shippingRate\": { \"@type\": \"MonetaryAmount\", \"value\": \"4.95\", \"currency\": \"GBP\" },\n \"deliveryTime\": {\n \"@type\": \"ShippingDeliveryTime\",\n \"handlingTime\": { \"@type\": \"QuantitativeValue\", \"minValue\": 0, \"maxValue\": 1, \"unitCode\": \"DAY\" },\n \"transitTime\": { \"@type\": \"QuantitativeValue\", \"minValue\": 2, \"maxValue\": 4, \"unitCode\": \"DAY\" }\n }\n}Returns can differ by country too; give each MerchantReturnPolicy its applicableCountry. See Organization and LocalBusiness data and policy pages AI assistants can quote.
<html lang> attribute matches the page's language.x-default.priceCurrency matches.shippingDetails where you can.If your country stores run on separate domains, such as northwind.example and northwind-uk.example, scan each one with AgentScore to compare them. To check a real journey in another market, such as \"find a rain jacket and get to checkout on the UK store\", run it with Ghost Agent and read the replay.
Ask an AI agent to buy \"the navy jacket in a medium\" and the hard part isn't finding the jacket. It's choosing navy and medium. Size, color and other variant pickers are where many product pages lose agents: the swatch has no name, the selected size looks different but reads the same, and the sold-out option can still be clicked. This guide covers how to build variant pickers AI agents can use without guessing.
\nA person sees a row of colored circles, notices a thicker border around one, and sees a crossed-out \"XL\". A browser agent mostly reads the page's structure: each control's role, its name and its state. If the swatch is a <div> with a background color, it has no role and no name. If \"selected\" is shown only with a border, there's no state to read. The agent may click the right circle by chance, click the wrong one, or not realize there's a choice to make at all.
The cost is direct. An agent that can't choose a size either gives up and tells the shopper it couldn't complete the task, or adds the wrong item to the cart. Both are lost or returned orders. And the same problems affect shoppers who use a keyboard or a screen reader, so fixing them helps people too.
\nAgents need four things from a picker: what each option is called, which one is chosen, which ones can't be bought, and confirmation that the page changed after they chose.
\nThe simplest, most reliable picker is a group of native radio buttons. Only one option can be chosen, the browser handles the keyboard, and every agent knows what a radio button is. You can style the label to look like a swatch or a size box.
\n<fieldset>\n <legend>Color</legend>\n <label class=\"swatch\">\n <input type=\"radio\" name=\"color\" value=\"navy\" checked>\n <span class=\"dot\" style=\"background:#1f2a44\" aria-hidden=\"true\"></span>\n Navy\n </label>\n <label class=\"swatch\">\n <input type=\"radio\" name=\"color\" value=\"olive\">\n <span class=\"dot\" style=\"background:#5b5e3a\" aria-hidden=\"true\"></span>\n Olive\n </label>\n</fieldset>aria-label. A hex code or an image file name isn't a name.<fieldset> with a <legend> (\"Color\", \"Size\") tells the agent what the choice is about.alt=\"\", so the name isn't read twice.If your design uses buttons rather than radios, that works too, as long as they are real <button> elements with names. AgentScore's checks that buttons and links have names, and that clickable things are real buttons and links, look for exactly these problems, though they run on your home page. A product page needs the same care.
Selection is the detail most often shown only visually. Radio buttons report it for free through checked. Custom buttons have to say it themselves:
<div role=\"group\" aria-labelledby=\"size-label\">\n <span id=\"size-label\">Size</span>\n <button type=\"button\" aria-pressed=\"false\">S</button>\n <button type=\"button\" aria-pressed=\"true\">M</button>\n <button type=\"button\" aria-pressed=\"false\">L</button>\n</div>aria-pressed=\"true\" on the chosen button and \"false\" on the others, and update them when the choice changes.role=\"radio\" with aria-checked inside a role=\"radiogroup\". It's more work to get the keyboard right, which is why native radios are usually the better choice.Hiding a sold-out size makes the shopper wonder whether it ever existed. Showing it with only a strike-through or faded color leaves an agent free to choose it, then fail at \"Add to cart\". Say it in words and in the control's state.
\n| Approach | What an agent sees |
|---|---|
| Option faded or struck through with CSS only | A normal option it can pick |
| Option removed from the page | No sign the size exists |
Native radio with disabled and \"XL, sold out\" in the label | An option that can't be chosen, and why |
Button with aria-disabled=\"true\" and \"XL, sold out\" in its name | The same, and it stays reachable by keyboard |
If you offer back-in-stock alerts, make that a separate, named button (\"Email me when XL is back\"), so an agent can tell it apart from buying.
\nMany AI assistants fetch pages rather than click through them. They can't press a swatch, but they can open a link. If each variant has its own address, an assistant can point a shopper straight to \"the navy jacket in medium\", and a browser agent can recover if a click doesn't take.
\nhttps://northwind.example/products/field-jacket?color=navy&size=m\nhttps://northwind.example/products/field-jacket?variant=40318Choosing a larger size or a different material often changes the price. If the new price only appears in an image, or changes quietly in a corner of the page, an agent may report the old one to the shopper.
\naria-live=\"polite\"), so the change is announced to assistive tech and is easier for agents reading the page structure to notice.Agents that read your page HTML rather than click it rely on structured data for the full list of options. schema.org has a ProductGroup type for this: one group, with variesBy naming the dimensions and hasVariant listing each variant as a Product with its own offer. Google documents the pattern in its product variant structured data guide.
{\n \"@context\": \"https://schema.org\",\n \"@type\": \"ProductGroup\",\n \"name\": \"Field jacket\",\n \"productGroupID\": \"FJ-100\",\n \"variesBy\": [\"https://schema.org/color\", \"https://schema.org/size\"],\n \"hasVariant\": [{\n \"@type\": \"Product\",\n \"sku\": \"FJ-100-NAV-M\",\n \"color\": \"Navy\",\n \"size\": \"M\",\n \"url\": \"https://northwind.example/products/field-jacket?variant=40318\",\n \"offers\": {\n \"@type\": \"Offer\",\n \"price\": \"189.00\",\n \"priceCurrency\": \"USD\",\n \"availability\": \"https://schema.org/InStock\"\n }\n }]\n}AgentScore's \"Product pages give price and stock in a form agents can read\" check reads offers inside ProductGroup and hasVariant, as well as on a plain Product. For more on the fields that matter, see product data AI shopping agents can read and prices AI agents can read.
Where to start. If your theme uses <div> swatches, switching to labelled radio buttons styled the same way usually fixes naming, selection state and keyboard use in one change. Ask your developer or theme provider for that first.
For the rest of the page's controls, see buttons, links and forms agents can use. For what happens after \"Add to cart\", see carts AI agents can manage.
", "date_published": "2026-10-09T13:06:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Navigability" ] }, { "id": "https://ghostagentlab.com/articles/infinite-scroll-pagination-ai-agents/", "url": "https://ghostagentlab.com/articles/infinite-scroll-pagination-ai-agents/", "title": "Infinite scroll, “load more” and pagination for AI agents", "summary": "Infinite scroll can hide most of your catalog from AI agents. Back it with real paginated URLs, a “load more” link and a sitemap.", "content_html": "Infinite scroll feels effortless to a shopper with a thumb. To many AI agents, it's a wall. Products that only load as someone scrolls may never be seen by an assistant that fetches the page, and a browser agent can waste its time scrolling and waiting. The fix isn't to drop infinite scroll or \"load more\". It's to put real, linked pages underneath them.
\nDifferent agents meet a category page in different ways:
\nSo a category of 300 products that shows 24 at a time, with more loading on scroll, can look like a category of 24. If the product a shopper asked for is number 61, the assistant may tell them you don't sell it.
\nThe foundation is the same for infinite scroll, \"load more\" and numbered pages: each chunk of results should exist at its own address, and that address should load those results when opened fresh.
\nhttps://northwind.example/collections/coffee\nhttps://northwind.example/collections/coffee?page=2\nhttps://northwind.example/collections/coffee?page=3?page=2 and /page/2 are separate URLs. #page=2 is not: servers never see what follows the #, so it can't return different results.?page=2 should arrive with its products already in it, not an empty shell that fills in later. See why AI agents can't see your JavaScript-only content.?roast=dark&sort=price-asc&page=2 should keep both. Our guide to site search and filters AI agents can use covers filter URLs in detail.Google's guidance on pagination and incremental page loading takes the same approach, and it works for AI agents for the same reasons.
\nUnique URLs only help if agents can find them. Each page should link to the next one, and ideally to the previous one and a few page numbers, with ordinary <a href> links in the HTML.
<nav aria-label=\"Pagination\">\n <a href=\"/collections/coffee?page=1\">Previous page</a>\n <a href=\"/collections/coffee?page=1\">1</a>\n <a href=\"/collections/coffee?page=2\" aria-current=\"page\">2</a>\n <a href=\"/collections/coffee?page=3\">3</a>\n <a href=\"/collections/coffee?page=3\">Next page</a>\n</nav><nav> with a name like \"Pagination\", so agents can tell it apart from the main menu.aria-current=\"page\".\"Load more\" is a good middle ground for shoppers, and it can work well for agents if it's built as two things at once: a real button for people and browser agents, and a real link for everyone else.
\n<a class=\"load-more\" href=\"/collections/coffee?page=3\">\n Load more products (264 remaining)\n</a>Start with a plain link to the next page. Then let JavaScript take over: intercept the click, fetch the next page, append the products, and update the address with history.pushState so the URL matches what's on screen. If the script doesn't run, or the agent doesn't run scripts, the link still goes to page three. If you prefer a <button>, keep the numbered page links alongside it so there's still a route.
If you want true infinite scroll, treat it as an enhancement over paginated pages. As each new batch loads, update the URL to the page number the shopper has reached. Keep the paginated links in the HTML, even if they're visually tucked away at the bottom. And make sure the footer is reachable: if the page grows every time someone nears the bottom, neither people nor agents can reach your contact, shipping and returns links.
\n| Pattern | Fetching assistants and crawlers | Browser agents |
|---|---|---|
| Infinite scroll only | See the first batch only | Must scroll and wait, can't tell when it ends |
| \"Load more\" button with no URL | See the first batch only | Can click, if the button has a name |
\"Load more\" link to ?page=2 | Can follow every page | Can click or follow |
| Numbered page links | Can follow every page | Can jump straight to a page |
For years, <link rel=\"next\"> and rel=\"prev\" in the page head were the standard way to tell search engines that pages belonged in a series. Google said in 2019 that it no longer uses them for indexing. Other search engines and tools may still read them, and they're part of HTML, so they do no harm. But don't rely on them: an agent following links needs the visible <a href> links in the page.
Two related head tags matter more:
\n?page=2 at page one tells crawlers to ignore the products on page two. See canonical URLs and duplicate pages.noindex by default. Pages that stay out of the index can, over time, get less attention from crawlers, and so can the products linked from them.Even with perfect pagination, some products sit deep in a long list. Your XML sitemap should list every product page directly, so crawlers don't depend on paging through categories to find them. AgentScore's \"Valid sitemap\" check confirms you have one. See XML sitemaps for AI agents. The sitemap is the safety net. The links are still the main route.
\n?page=2 directly in a private window. It should show the second set of products, not the first and not an empty page.Test it as a shopper would. Ask a Ghost Agent to \"find the cheapest dark roast in the coffee category\" on a category of more than one page. The step-by-step replay shows whether it reached the later pages or settled for what loaded first.
\nFor the bigger picture of how agents move around a site, see What is agent readiness?
", "date_published": "2026-10-09T13:05:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Navigability" ] }, { "id": "https://ghostagentlab.com/articles/error-messages-ai-agents/", "url": "https://ghostagentlab.com/articles/error-messages-ai-agents/", "title": "Error messages AI agents can understand and fix", "summary": "Help AI agents recover from mistakes: error text tied to each field, live regions, useful 404 pages and HTTP status codes that match what the page says.", "content_html": "Every journey hits an error eventually: a mistyped postcode, a promo code that's expired, a product that's gone. A person reads the red text, fixes it and carries on. An AI agent can only do the same if the error is written down, attached to the right field and reported honestly by your server. Otherwise it tries again, gives up, or tells the shopper something went wrong without saying what.
\nAn agent working for someone wants to finish the task. When a form doesn't go through, it needs to answer three questions: did something go wrong, which field caused it, and what should it enter instead? People answer those from visual cues, such as a red border, a shake, an icon or a message that flashes up and disappears. Agents mostly read the page's text and structure, and may not catch any of those cues.
\nA clear error turns a failed attempt into a quick correction. An unclear one ends the journey, and you lose the order or sign-up without anyone telling you. The same fixes help people who use screen readers, and anyone filling in a form on a small screen.
\nStart with the words. An agent passes the message on, or acts on it, exactly as written.
\n| Unhelpful | Helpful |
|---|---|
| Invalid input | Enter a 5-digit ZIP code, like 10001 |
| Error | This email address is already registered. Log in or reset your password. |
| Promo code not applied | The code AUTUMN10 expired on September 30 |
| Something went wrong | We couldn't save your address. Please try again in a minute. |
An error floating near a form isn't enough. Agents, like screen readers, need to know which field it belongs to. Two attributes do that:
\n<label for=\"zip\">ZIP code</label>\n<input id=\"zip\" name=\"zip\" autocomplete=\"postal-code\"\n aria-invalid=\"true\" aria-describedby=\"zip-error\">\n<p id=\"zip-error\">Enter a 5-digit ZIP code, like 10001</p>aria-invalid=\"true\" marks the field as having a problem. Remove it once the value is fixed.aria-describedby links the field to its message, so the message is read as part of the field's description.required attribute, and the right type and autocomplete values, so agents can get it right first time. Labels matter here too: see buttons, links and forms agents can use.On a long form, such as checkout or account sign-up, add a short summary at the top when it's submitted with problems. It lists each error as a link to its field, and gets keyboard focus so it's the first thing read.
\n<div role=\"alert\" tabindex=\"-1\" id=\"error-summary\">\n <h2>There are 2 problems with your details</h2>\n <ul>\n <li><a href=\"#zip\">Enter a 5-digit ZIP code, like 10001</a></li>\n <li><a href=\"#phone\">Enter a phone number with area code</a></li>\n </ul>\n</div>For messages that appear without a page load, such as \"Promo code applied\" or \"Only 2 left, quantity updated\", use a live region. role=\"alert\" is for urgent errors. role=\"status\" or aria-live=\"polite\" is for confirmations. Keep the message on screen until the next action: toasts that fade after three seconds are easy for an agent to miss.
Errors aren't only in forms. When an agent follows an old link or a product is discontinued, your error page is what it reads. It should explain, and offer a way forward.
\navailability to OutOfStock in your product data. Don't turn them into errors.Agents and crawlers read your HTTP status code before they read a word of the page. If they disagree, the code usually wins. A \"Page not found\" message sent with a 200 OK status, often called a soft 404, tells crawlers the page is fine, so the dead page may stay in AI search answers. A real product page that returns an error tells them it's gone.
| Situation | Status code |
|---|---|
| Page loaded normally, including out-of-stock products | 200 |
| Page moved for good | 301 or 308 to the new address |
| Page doesn't exist | 404 |
| Page removed on purpose and not coming back | 410 |
| Too many requests | 429, with a Retry-After header |
| Something broke on your side | 500 |
| Down for maintenance or overloaded | 503, with a Retry-After header |
Two common mistakes: redirecting every missing page to the home page, which leaves agents on a page that doesn't answer the question, and serving a block page with 200, which looks like content. The meanings of these codes are defined in RFC 9110. For 429s and agents, see rate limits for AI agents.
For form submissions, what matters most is that the page sent back shows the errors in text, tied to their fields. Whether you re-render the form with 200 or use 422 is less important than making the message clear.
AgentScore doesn't have a check for error messages, since it never submits your forms. Several checks cover the groundwork. \"Form fields are labelled\" checks the home page's fields, and \"Cart and checkout fields are labelled for agents\" checks the cart and checkout. \"AI agents can open your product, pricing and cart pages\" compares the status codes AI assistants get with what a browser gets, which catches bot protection sending agents errors that people never see.
\nTo see what happens when an agent actually makes a mistake, run a journey. A Ghost Agent types the test details you give it and records each step, so if a form rejects them without saying why, the replay shows exactly where it stalled. Ghost Agents stop at the payment form, so payment errors are out of scope. To find the errors agents hit in real traffic, see finding the errors AI agents hit in your logs.
\naria-invalid=\"true\" and aria-describedby pointing to its message.404 or 410, with a search box and useful links.200 status.503 or 429 with Retry-After.Try it yourself. Submit your checkout's address form with a wrong ZIP code, then read only the text on the page. If you can't tell which field is wrong and what it wants, neither can an agent.
\nFor more on the forms themselves, see how to make checkout work for AI shopping agents and sign-up, booking and lead forms AI agents can finish.
", "date_published": "2026-10-09T13:04:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Navigability" ] }, { "id": "https://ghostagentlab.com/articles/cart-ai-agents/", "url": "https://ghostagentlab.com/articles/cart-ai-agents/", "title": "Carts AI agents can manage: quantities, promo codes and totals", "summary": "How to build a cart AI agents can change and check: named quantity and remove controls, usable promo code fields, clear totals and carts that last.", "content_html": "The cart is where an AI agent checks its work. A person asks for \"two bags of the decaf, and use my code SPRING10\", and the agent has to change a quantity, apply a code and read back a total the person will agree to pay. If the cart makes any of that hard, the agent guesses or gives up. This guide covers what a cart needs so agents can manage it reliably.
\nA person glances at a cart and knows whether it looks right. An agent has to read it. Before it hands over for payment, a careful agent reports back: what's in the cart, how many, what it costs and when it arrives. It gets every one of those facts from your cart page. If the cart can't be read, the agent can't confirm, and the order stalls at the point where you've already done the hard work of winning it.
\nFor the whole journey from search to payment, see how to make checkout work for AI shopping agents. This guide zooms in on the cart itself.
\nThe most common problem is a pair of \"−\" and \"+\" icons with no names, next to a number that's styled text rather than a field. A person sees a stepper. An agent sees two unnamed buttons and a number it can't change directly.
\n<input type=\"number\"> whose label names the item lets an agent type the quantity it wants.A cart line that does all of this:
\n<li>\n <a href=\"/products/ethiopia-yirgacheffe?variant=340g\">Ethiopia Yirgacheffe, 340 g</a>\n <label for=\"qty-1\">Quantity</label>\n <input id=\"qty-1\" name=\"qty-1\" type=\"number\" min=\"1\" max=\"4\" value=\"2\">\n <span>Limit 4 per order</span>\n <span>$36.00</span>\n <button type=\"button\" aria-label=\"Remove Ethiopia Yirgacheffe, 340 g\">\n <svg aria-hidden=\"true\">…</svg>\n </button>\n</li>\n<p role=\"status\">Quantity updated to 2. Subtotal $36.00.</p>AgentScore's \"Buttons and links have names\" check looks for exactly this kind of icon-only control on your home page. Apply the same rule in the cart, where it matters more.
\nPeople often ask an assistant to use a code they already have. That only works if the agent can find the field, apply the code and tell whether it worked.
\naria-expanded), not a styled line of text.The goal is simple: an agent should be able to say \"The total is $44.18, including $5.00 shipping and $3.18 tax\" and be right. That means each part of the total is on the page as text.
\n| Line | What to show |
|---|---|
| Subtotal | The sum of the items, as text, updated after every change |
| Discounts | Each code or promotion by name, with its amount |
| Shipping | The cost for the chosen method, or an estimate with the method named. If it truly depends on the address, say \"Calculated at checkout\" |
| Tax | The amount, or \"Calculated at checkout\". Say whether prices already include tax |
| Total | The final figure with its currency |
Most agents hand back to the person to sign in or pay, and that can happen hours later, or on another device. A cart that empties in the meantime undoes the agent's work.
\n/cart, as a full page as well as any slide-out drawer.An upsell window after \"Add to cart\", or on the way to checkout, is one more thing an agent has to get past.
\nThe free AgentScore scan is read-only: it loads pages but never adds anything to a cart. On many stores the cart is empty when it looks, and the scan says so rather than guessing. For stores, these checks cover the cart:
\nChanging quantities, applying codes and reading totals need items in the cart, so they belong to Task completion, which the free scan marks \"not tested\". To test them, set up a Ghost Agent test with a goal in your own words, such as \"Add two bags of decaf, apply the code TEST10 and go to checkout\", using a test code you've created, and a pass condition that looks for the code's name on the final page. Ghost Agents stop at the payment form, so no order is placed. See how to test your key journeys with AI agents.
\nA five-minute cart test. Add two items, then use only the keyboard to change a quantity, remove an item and apply a code. Then read the cart aloud as if over the phone: items, quantities, discount, shipping, tax, total. Anything you had to look at a picture to work out, an agent can't see.
\nBuying is only part of what people ask AI assistants to do. \"Where's my order?\" \"Return the shoes, they're too small.\" \"Order the same coffee again.\" These after-sale tasks are frequent, they cost you nothing when customers can do them on their own, and they're often locked behind a login an agent can't use. This guide covers how to make them work for AI agents, and where the agent should hand back to the person.
\n\"Where is my order?\" is one of the most common reasons customers contact an online store. When an assistant can answer it from your site, that's a support ticket that never gets written. When it can't, the person contacts you anyway, or the assistant pieces together an answer from a carrier's site and gets it wrong.
\nReturns matter before the sale, too. Assistants read return policies when comparing stores, and a clear, easy returns process is part of what makes a store safe to recommend. See policy pages AI assistants can quote.
\nAn agent can't sign in as the person, but it can usually be given an order number and an email address. That's enough for a well-built order lookup.
\n/orders/lookup, linked from your footer and help pages with an ordinary link.A lookup form agents can fill in:
\n<form action=\"/orders/lookup\" method=\"post\">\n <label for=\"order\">Order number</label>\n <input id=\"order\" name=\"order\" autocomplete=\"off\" aria-describedby=\"order-hint\">\n <p id=\"order-hint\">Starts with NW-. You'll find it in your confirmation email.</p>\n <label for=\"email\">Email used at checkout</label>\n <input id=\"email\" name=\"email\" type=\"email\" autocomplete=\"email\">\n <button type=\"submit\">Find my order</button>\n</form>Show only what's needed on a page reached this way: status, items, carrier and the delivery city. Leave out the full address, phone number and payment details.
\n\"Order the same again\" usually needs order history, and order history usually needs an account. You can still make it easy:
\nThese tasks touch personal details and money, so there's a real tension. Agents shouldn't need the person's password, and anyone holding an order number shouldn't be able to change an order. The answer is to match the check to the risk:
\n| Task | What it should take | Who finishes it |
|---|---|---|
| Track an order | Order number and email | The agent |
| Check return eligibility and options | Order number and email | The agent |
| Submit a return or exchange | Order number and email, plus a review step | The agent, once the person agrees |
| Reorder | A \"Buy again\" link or button | The agent builds the cart, the person pays |
| Change a delivery address or cancel | A one-time code or emailed link | The person |
| Change email, password or saved cards | Full sign-in, ideally with two-step sign-in | The person only |
The same thinking applies to buying: see guest checkout and sign-up and booking forms agents can use.
\nStart with tracking. It's the most common after-sale question, it needs no new security thinking, and a public lookup page with a direct link in every order email works for customers and their agents alike.
\nAgent readiness work needs a monthly number, or it quietly slides down the priority list. This guide sets out a one-page monthly report built on six measures: where each one comes from, what good looks like, and how to read them together. There are no reliable industry benchmarks for most of these yet, so the comparison that matters is your own trend.
\n| Measure | Question it answers | Where it comes from |
|---|---|---|
| AgentScore | Can agents get in, read and find their way around? | An AgentScore scan |
| Agent visits by type | Who's visiting, and what are they reading? | Server or CDN logs |
| Verification share | How much of that traffic is really who it says it is? | Logs checked against operators' published proof |
| Blocks and errors | Are we turning agents away? | Status codes in your logs |
| Ghost Agent pass rates | Can an agent actually finish the job? | Ghost Agent tests |
| AI referral visits and orders | Is it bringing in customers and revenue? | Your analytics and your store's orders |
In Ghost Agent Labs, the site's Overview page brings several of these together. The sections below name the page that holds each one.
\nReport: the overall score and its band, the scores for Access, Readability and Navigability, and how many findings you fixed and how many are new. The free scan doesn't test Task completion yet, so the score covers the other three categories.
\nWhere: the Readiness page. Every scan is saved, so you can track the score over time, and \"All findings\" exports to CSV for your developers.
\nGood looks like: a steady or rising score, no critical findings open for more than a month, and every drop explained. Scan on the same day each month and after major releases, so the numbers compare like with like. The bands are 80 and up Good, 60 to 79 Fair, 40 to 59 Weak and below 40 Poor. See reading your AgentScore report.
\nReport: requests by visitor type (search crawlers, AI training crawlers, AI assistants and autonomous agents) compared with last month, the pages agents read most, and any new agents.
\nWhere: the Agent traffic page (\"Requests by visitor type\", \"Agent sessions\" and \"Pages agents request most\"), and \"What changed\" on the Overview, which lists first-time agents and agents whose traffic rose or fell by half or more.
\nGood looks like: AI assistant and AI search traffic holding steady or growing, and agents reaching product, pricing and policy pages rather than stalling on the home page or old URLs. Don't celebrate total volume. A busy training crawler isn't a customer. Assistant requests are the closest thing to a customer visit from AI, because each one is usually a person's question. See how to measure AI agent traffic.
\nReport: of the requests claiming to be a known agent, the share that were verified, couldn't be verified, and were spoofed.
\nWhere: the Verification & spoofing page.
\nGood looks like: spoofed traffic close to zero, or falling after you block it, and a verified share that rises as more operators publish ways to check their agents. This measure also keeps the rest of the report honest: never report \"ChatGPT visited 40,000 times\" from user agent names alone, because impostors use those names too. See how to tell if an AI crawler is real.
\nReport: the share of agent requests that were blocked (401, 403 and 429 responses), the 404 and server error rates, and the agents and pages most affected.
\nWhere: \"Errors: agents vs people\" on the Overview, \"What agents got back\" on the Agent traffic page, and the status codes on each agent's page.
\nGood looks like: blocked responses to verified AI assistants near zero, agents getting errors about as often as people do rather than much more, and agent 404s trending down as you redirect old URLs. A sudden rise almost always follows a change to bot protection or rate limits. See finding the errors AI agents hit.
\nReport: for each journey you test, such as checkout, product search or sign-up, whether it's passing, flaky or failing, its pass rate, and the step where failures cluster.
\nWhere: the Ghost Agents page, with a replay of every run.
\nGood looks like: your money journeys passing, flaky results treated as findings rather than noise, and failures fixed within a sprint. A pass rate below 100% means some customers' assistants would fail too. This is the measure closest to revenue, because it tests whether an agent can do the job, not just whether the page looks right. See how to test your key journeys with AI agents.
\nReport: visitors who arrived from AI assistants' answers, by assistant, their conversion rate, and orders and revenue from AI.
\nWhere: your analytics, set up as in tracking visits and sales from AI assistants. In Ghost Agent Labs, \"Visitors from AI assistants\" is on the Overview. Send your store's orders to the site's orders URL and Mission Control shows \"Agent-driven revenue\": orders referred by an AI answer or placed by an agent, and their share of revenue.
\nGood looks like: growth from whatever base you start at, and conversion from AI referrals that holds up against your other referral sources. Present it as a floor, not a total: app traffic and copied links arrive as \"direct\", and many AI answers influence a purchase without a click.
\nSingle numbers mislead. These combinations are where the report earns its keep:
\nWhat to leave out. Total bot requests without a breakdown, scores without a trend, and industry comparisons nobody can source. They invite the wrong questions. If you're presenting upward, see how to explain agent readiness to your board, and for what to fix first, the 90-day agent readiness plan.
\nWhen an AI agent can't get into your site, nobody tells you. There's no complaint and no support ticket. The assistant just answers with someone else's products. The evidence is in your server and CDN logs. This guide shows which errors to look for, how to find them with a few queries, and how to separate real agents from impostors before you change anything.
\nIt builds on how to measure AI agent traffic, which covers getting hold of your logs in the first place.
\n| What you see | What it usually means for an agent | Who usually fixes it |
|---|---|---|
| 403 Forbidden | Bot protection, a firewall rule or a country block turned it away | Bot protection or CDN admin |
| 401 Unauthorized | The page needs a login | E-commerce or product team |
| 429 Too Many Requests | It hit a rate limit | CDN admin or developers |
| Challenge page | A \"checking your browser\" page or CAPTCHA it can't solve. Often a 403 or 503, sometimes a 200 | Bot protection or CDN admin |
| 500, 502, 503 | Your server failed. Agents can reach pages people rarely visit, which may not be cached | Developers or hosting |
| 504, slow responses, 499 | The page took too long. In nginx, 499 means the client gave up and closed the connection first | Developers or hosting |
| 404 Not Found | An old or guessed URL. AI models often remember URLs you've since changed | Content or SEO team |
Challenge pages are the easiest to miss, because some bot protection serves them with a 200 status, so they look like successes. Look for your CDN's security or bot action field instead. Cloudflare, for example, adds a cf-mitigated: challenge header to challenge responses. A response that's much smaller than the real page usually is one too.
Most logs already have the basics. Check yours include:
\nOn nginx, a log format like this captures them:
\nlog_format agents '$time_iso8601 $remote_addr \"$request_method $uri\" '\n '$status $body_bytes_sent $request_time \"$http_user_agent\"';If you only have the standard \"combined\" access log, this one-liner counts status codes for three AI assistants:
\ngrep -E 'ChatGPT-User|Claude-User|Perplexity-User' access.log \\\n | awk '{print $9}' | sort | uniq -c | sort -rnOnce logs are in a database or log tool, a few queries answer most questions. The examples below are illustrative, written for PostgreSQL against a table called requests with columns ts, client_ip, user_agent, path, status, bytes, duration_ms and security_action. Rename them to match your own log store; the same ideas work in BigQuery, Athena or a log search tool.
SELECT\n CASE\n WHEN user_agent ILIKE '%ChatGPT-User%' THEN 'ChatGPT-User'\n WHEN user_agent ILIKE '%OAI-SearchBot%' THEN 'OAI-SearchBot'\n WHEN user_agent ILIKE '%Claude-User%' THEN 'Claude-User'\n WHEN user_agent ILIKE '%Perplexity-User%' THEN 'Perplexity-User'\n WHEN user_agent ILIKE '%PerplexityBot%' THEN 'PerplexityBot'\n WHEN user_agent ILIKE '%Googlebot%' THEN 'Googlebot'\n ELSE 'everything else'\n END AS agent,\n COUNT(*) AS requests,\n COUNT(*) FILTER (WHERE status IN (401, 403)) AS blocked,\n COUNT(*) FILTER (WHERE status = 429) AS rate_limited,\n COUNT(*) FILTER (WHERE status = 404) AS not_found,\n COUNT(*) FILTER (WHERE status >= 500) AS server_errors,\n ROUND(100.0 * COUNT(*) FILTER (WHERE status >= 400) / COUNT(*), 1) AS error_pct\nFROM requests\nWHERE ts >= now() - interval '7 days'\nGROUP BY 1\nORDER BY requests DESC;\"Everything else\" mixes people with other bots, many of them scanners probing for pages that don't exist. For a fair baseline, compare with requests from ordinary browsers: if AI agents get errors far more often than people do, something is turning them away.
\nSELECT path, status, COUNT(*) AS hits\nFROM requests\nWHERE ts >= now() - interval '7 days'\n AND user_agent ~* '(ChatGPT-User|Claude-User|Perplexity-User)'\n AND (status IN (401, 403, 429) OR security_action = 'challenge')\nGROUP BY path, status\nORDER BY hits DESC\nLIMIT 20;SELECT date_trunc('day', ts) AS day,\n COUNT(*) FILTER (WHERE security_action = 'challenge') AS challenged,\n COUNT(*) FILTER (WHERE status IN (499, 504)) AS timed_out,\n COUNT(*) FILTER (WHERE duration_ms > 10000) AS over_10s\nFROM requests\nWHERE user_agent ~* '(ChatGPT-User|Claude-User|Perplexity-User|OAI-SearchBot)'\nGROUP BY 1\nORDER BY 1;A step change on one day usually lines up with a release or a bot protection change. That's your first lead.
\nEverything above trusts the user agent, and a user agent is only a claim. Scrapers often call themselves ChatGPT-User or Googlebot to get past bot protection. When your firewall blocks them, that 403 is working as intended. So before you loosen any rule, split agent errors into three groups:
\nOperators such as OpenAI publish their agents' IP ranges (see OpenAI's crawler documentation). Load them into a table, refresh it regularly, and join:
\n-- agent_ranges(agent text, cidr cidr), loaded from each operator's published list\nSELECT r.agent IS NOT NULL AS verified, l.status, COUNT(*) AS hits\nFROM requests l\nLEFT JOIN agent_ranges r\n ON l.client_ip::inet << r.cidr\n AND l.user_agent ILIKE '%' || r.agent || '%'\nWHERE l.user_agent ~* '(ChatGPT-User|OAI-SearchBot)'\n AND l.ts >= now() - interval '7 days'\nGROUP BY 1, 2\nORDER BY 1 DESC, hits DESC;For the methods in detail, including reverse DNS and signed requests, see how to tell if an AI crawler is real.
\nMind the privacy. Logs hold IP addresses, which count as personal data for human visitors. Filter to agent user agents before you keep detailed records, hash or drop IP addresses for everyone else, and keep a retention period that matches your privacy policy.
\nRetry-After. See rate limits for AI agents.If writing queries isn't your team's idea of a good week, Ghost Agent Labs does this from your Cloudflare, Vercel or JSON logs. The Agent traffic page shows what agents got back, each agent's page shows its status codes, the Verification & spoofing page splits verified from spoofed, and Alerts tell you when a major AI assistant starts being blocked. Either way, check after every bot protection or CDN change, and at least monthly as part of your agent readiness KPIs.
", "date_published": "2026-10-09T13:00:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Measurement" ] }, { "id": "https://ghostagentlab.com/blog/seo-to-agent-readiness/", "url": "https://ghostagentlab.com/blog/seo-to-agent-readiness/", "title": "SEO got you found. Agent readiness gets you chosen", "summary": "SEO decides whether AI agents find you. Agent readiness decides whether they can use your site and choose you. What carries over, and what's new.", "content_html": "For twenty years, SEO has answered one question: when someone searches, do they find you? AI agents add a second question that SEO was never designed to answer. When an agent arrives on someone's behalf, can it actually use your site? Getting found is still the start. Getting chosen now depends on what happens next.
\nSearch engines crawl, index and rank. A person sees a list of results, clicks one, and does the rest themselves. Everything after the click is the person's job: reading the page, closing the cookie banner, finding the size selector, getting through checkout. If the site is awkward, people usually push through.
\nSEO teams have become very good at the part before the click. Titles, descriptions, sitemaps, structured data, internal links, page speed, crawl budgets. All of that still matters.
\nWhen someone asks an AI assistant to compare three running shoes, check which one is in stock in a size 10, and buy it, there is no results page and often no click by a person. The assistant fetches pages itself. A browser agent may open your site, search your catalog and try to add to cart.
\nThat shifts the work from the person to the agent, and agents don't push through. If the price isn't readable, the agent can't quote it. If a pop-up has no labelled close button, the journey ends there. If bot protection challenges the assistant's request, the person gets a competitor's answer instead of yours. Nobody tells you. The visit simply doesn't turn into anything.
\nSo the question moves from \"do we rank?\" to \"when an agent arrives, can it get in, understand the page, find its way and finish the job?\" That's what we mean by agent readiness.
\nThe good news for SEO teams is that a lot of the groundwork is shared. Several AgentScore checks will look familiar.
\n| SEO practice | What it does for AI agents |
|---|---|
| A clean robots.txt | The same file controls AI assistants, AI search and training crawlers, each by its own name. See robots.txt for AI agents. |
| Clear titles, descriptions and headings | How an agent decides whether a page answers the question it was sent to ask. See titles, descriptions and headings AI agents understand. |
| Product structured data | Lets an assistant quote your price and stock correctly instead of guessing. See product data AI shopping agents can read. |
| A valid XML sitemap | Helps agents find pages that aren't linked prominently. See XML sitemaps for AI agents. |
| Image alt text | The only way an agent reading text knows what a product photo shows. See alt text for AI agents. |
If your SEO is in good shape, you have a head start. But it's a head start, not the finish.
\nSEO teams check that Googlebot can crawl. They rarely check how bot protection treats ChatGPT-User, Perplexity-User or a browser agent. A site can rank well and still challenge every AI assistant at the door, because the rules that protect it from scrapers were tuned for a world where the only good bots were search engines. See bot protection and CAPTCHAs.
\nGoogle renders JavaScript, so a page built entirely in the browser can still rank. Most AI crawlers and many assistants don't run JavaScript. They read the HTML your server sends, and if the price or description only appears after scripts load, they see an empty page. See why AI agents can't see JavaScript-only content and prices AI agents can read.
\nSEO ends at the landing page. Agent readiness carries on through the page: buttons and links with names, form fields with labels, pop-ups that can be closed, and an add-to-cart button an agent can find and press. These look like accessibility concerns because they largely are. See accessibility work is agent readiness work.
\nThe biggest difference is the last step. An agent sent to buy something either completes the purchase or it doesn't. No ranking report tells you which. The only way to know is to send a real agent through the journey and watch. That's what Ghost Agents do, and it's why Task completion carries the most weight in AgentScore.
\nSEO is measured in rankings, impressions and organic sessions. Agents mostly don't run your analytics tag, so they don't show up in those numbers. Agent visits live in server and CDN logs, and the visits and sales that AI assistants refer need their own tracking. See how to measure agent traffic and tracking visits and sales that come from AI assistants.
\nHere's the simplest way to put the difference to a leadership team.
\nAn assistant that can't read your stock level has no reason to recommend you over a competitor whose stock level is right there in the HTML. Being found and then failing the agent is close to not being found at all.
\nThis isn't a new discipline that replaces SEO. It's SEO's natural next step, and SEO teams are well placed to lead it. They already own robots.txt, titles, sitemaps and much of the content. They already work with developers on structured data and rendering. What's new is a set of partners SEO hasn't always needed: whoever runs bot protection and the CDN, and whoever owns checkout. We cover that split in who owns agent readiness?
\nThree things an SEO lead can do this week:
\nIf you want a structured path from there, our 90-day agent readiness plan breaks the work into three phases.
", "date_published": "2026-10-09T12:22:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Perspective" ] }, { "id": "https://ghostagentlab.com/blog/how-agentscore-works/", "url": "https://ghostagentlab.com/blog/how-agentscore-works/", "title": "How AgentScore works: what's behind the 100 points", "summary": "How AgentScore turns its checks into a score out of 100: the four categories, how points are earned, which pages it visits, and what isn't scored yet.", "content_html": "AgentScore gives your website one number out of 100 for how well AI agents can get in, read your pages and find their way around. A single number is only useful if you trust it, so here is exactly how it's worked out: the four categories, how each check earns points, which pages the scanner visits, and what it doesn't score yet.
\nEvery check belongs to one of four categories. Each category has a fixed number of points, set by how much it matters to an AI agent trying to do something for a customer.
\n| Category | Points | The question it answers |
|---|---|---|
| Access | 25 | Can AI agents get in, past robots.txt, bot protection and challenges? |
| Readability | 25 | Can they understand your pages without guessing? |
| Navigability | 20 | Can they find and use the buttons, links and forms they need? |
| Task completion | 30 | Can an AI agent actually finish a real task, such as reaching checkout? |
The background to these categories is in our guide to agent readiness. This post is about the mechanics.
\nEach check ends in one of four results:
\nWithin a category, checks carry different weights, because some problems stop an agent cold and others only slow it down. A category's score is its points multiplied by the share of weight you earned among the checks that could be tested. If Readability's tested checks are worth 17 in total and you earn 13.5 of them, you get 25 × 13.5 ÷ 17, or 19.9 out of 25.
\nThe overall score adds up the categories that could be tested and scales the result to 100. So if Task completion wasn't tested and you scored 20 for Access, 17 for Readability and 11 for Navigability, your AgentScore is 48 out of a possible 70, which is 69.
\nMost checks are rule-based rather than judged by an AI model, so the same site gets the same score from one scan to the next. When it moves, something on the site changed.
\n| Score | Band | What we tell you |
|---|---|---|
| 80 and up | Good | AI agents can use this site well. |
| 60 to 79 | Fair | AI agents can mostly use this site, but some things get in their way. |
| 40 to 59 | Weak | AI agents will struggle on this site. |
| Below 40 | Poor | Most AI agents will fail on this site. |
The one-line verdict at the top of a report also names your biggest issue: the failed check with the highest weight.
\nHere is what each category looks at today. The weight is how much the check counts inside its category.
\nWeights can change between scoring versions as we learn what trips agents up. Every report records the scoring version it used, so you can compare like with like.
\nAgentScore reads public pages only. It never adds anything to a cart, submits a form, signs up or places an order. In one scan it looks at:
\n/pricing and /cart. Each is fetched as a browser and as an AI assistant, and product and pricing pages are rendered in a browser too./checkout, fetched as a browser and as three AI assistants.Requests under the scanner's own name are signed so you can verify them. The requests that imitate AI agents are not, on purpose. We explain why in Why our scanner sometimes wears other agents' names. How it identifies itself and how to opt out are on the AgentScore bot page.
\nTask completion. The automated scan doesn't run an AI agent through a task yet, so \"An AI agent completes a real task\" shows as not tested and your score covers Access, Readability and Navigability. In the Ghost Agent Labs app, Ghost Agent tests already send real AI agents through your journeys. Their results don't feed into AgentScore yet.
\nStore-only checks on other sites. Guest checkout, checkout fields, product data and agentic commerce only apply to stores. If we find no products or cart, they're marked not tested.
\nAnything behind a full cart. Many checkouts only open with items in the cart. Because the scanner never adds any, those checks often say \"not tested\" rather than guess.
\nYour choice on training crawlers. Blocking AI training is a business decision, not a readiness problem, so it costs no points.
\nThen go beyond the score: set up tests of your key journeys with AI agents, so you know when a release breaks them.
", "date_published": "2026-10-09T12:21:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Product" ] }, { "id": "https://ghostagentlab.com/blog/introducing-ghost-agents/", "url": "https://ghostagentlab.com/blog/introducing-ghost-agents/", "title": "Ghost Agents: testing your site with real AI agents", "summary": "Ghost Agents are real AI agents that run your checkout, search and sign-up journeys on a schedule, with pass rates, step-by-step replays and alerts.", "content_html": "A readiness score tells you what might trip an AI agent up. It can't tell you whether one actually gets from your home page to checkout. Ghost Agents do. They're real AI agents that work through your key journeys the way a customer's assistant would, on a schedule, and show you every step.
\nChecks for labelled buttons, structured data and bot protection catch the common problems. But journeys fail in ways no rule predicts: a size picker the agent can't operate, a discount pop-up that appears on the second page, a checkout button that only shows after scrolling. People get past these without noticing. Agents often don't.
\nThis is what our team has done for human visitors for years with Ghost Inspector: prove the journeys that make you money still work, and find out when a release breaks one. Ghost Agents bring the same idea to AI agents.
\nYou write a goal in plain English, the way a customer might ask an AI assistant. To make it quick, there are ready-made templates for the journeys that matter most:
\n| Template | What the agent tries to do |
|---|---|
| Checkout | Find a popular product, add it to the cart, and check out as a guest up to the payment step |
| Product search | Use site search to find a product a customer might ask for, and open its page |
| Sign-up | Create an account with a test email address, as far as the site allows without email confirmation |
| Contact form | Find the contact or support form and fill it in with a short test question, without submitting it |
On each run, the agent opens your site in a fresh browser and repeats a simple loop. It looks at the page as an agent sees it: the address, the visible text, and the buttons, links and fields with their names. It picks one action, such as click, type, choose an option, scroll, go back or finish. Then it acts, and looks again.
\nAn agent saying \"done\" isn't proof. So you choose pass conditions that are checked against the final page the agent reaches, without asking an AI model:
\n/checkout.AI agents don't behave the same way every time, so each check runs the journey several times (three is a good default) and reports a pass rate. A test where every run passes is passing. One where none pass is failing. One where results disagree is flaky: agents can do the journey, but not reliably, which for a customer's assistant can mean the same as not at all.
\nTests run when you click \"Run now\", every day, or every hour. When one fails, you can be alerted by email or Slack.
\nEvery run is recorded. The replay opens on the step where things went wrong, with a screenshot of what the agent saw, the action it took, and its reasoning, which often says in plain words what it couldn't find or press. If a journey used to work, you can compare the failed run with the last passing one to see what changed.
\nThat turns \"agents can't check out\" into something a developer can act on: \"the cookie banner covers the checkout button and has no labelled close button\".
\nA test agent that clicks around a live store has to be careful. Ghost Agents follow hard rules that the model can't talk its way past:
\nThey also identify themselves. Every request carries GhostAgent/1.0 (+https://ghostagentlab.com/ghost-agent) at the end of a normal Chrome user agent and is signed with Web Bot Auth, so your CDN can verify it's really us. We explain signing in Why we sign every request our agents send, and the Ghost Agent page covers how to allow or opt out.
We'd rather you knew the limits up front:
\nFor help choosing journeys and writing good goals, read How to test your key journeys with AI agents.
", "date_published": "2026-10-09T12:20:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Product" ] }, { "id": "https://ghostagentlab.com/blog/accessibility-is-agent-readiness/", "url": "https://ghostagentlab.com/blog/accessibility-is-agent-readiness/", "title": "Accessibility work is agent readiness work", "summary": "Many browser agents read pages through the accessibility tree. Where WCAG and agent readiness overlap, where they differ, and how to make fixes pay twice.", "content_html": "If your team has spent time making your site work with screen readers and keyboards, you've already done a large share of the work to make it work for AI agents. The reason is simple: many browser agents read a page through the same structure that assistive technology uses.
\nA person looks at a page and sees layout, color and images. A browser agent, the kind that opens your site in a real browser to search, fill in forms and add to cart, needs something it can reason about and act on. It has two main options.
\nThe tooling used to build agents reflects this. Playwright's MCP server, a widely used way to give an AI model control of a browser, hands the model an accessibility snapshot of the page rather than pixels. In that snapshot, a button is a line like button \"Add to cart\". A button with no name is just button, and the agent has to guess.
<!-- What the agent can use -->\n<button type=\"submit\">Add to cart</button>\n<button aria-label=\"Open cart\"><svg aria-hidden=\"true\">…</svg></button>\n<!-- What it can't -->\n<div class=\"btn\" onclick=\"addToCart()\"><svg>…</svg></div>The last example has no role and no name. A screen reader user can't use it, and an agent reading the accessibility tree may not know it exists.
\nThe Web Content Accessibility Guidelines (WCAG) 2.2 are the standard most accessibility programs work to. Several of their success criteria line up closely with AgentScore checks.
\n| WCAG 2.2 success criterion | Related AgentScore check |
|---|---|
| 4.1.2 Name, Role, Value | Buttons and links have names; Clickable things are real buttons and links |
| 1.3.1 Info and Relationships, 3.3.2 Labels or Instructions | Form fields are labelled; Cart and checkout fields are labelled for agents |
| 1.1.1 Non-text Content | Images have text descriptions |
| 2.4.1 Bypass Blocks, 1.3.1 Info and Relationships | Page has main and navigation landmarks |
| 2.4.2 Page Titled, 2.4.6 Headings and Labels | Clear page title, description, and headings |
| 2.1.2 No Keyboard Trap, 2.4.11 Focus Not Obscured (Minimum) | Pop-ups and banners can be dismissed by agents |
The match isn't exact. WCAG asks more of each criterion than our checks test for, and our checks look at things from an agent's point of view. But the direction is the same: give every control a role and a name, label every field, describe every meaningful image, and structure the page so it can be navigated without seeing it.
\nOur guide to buttons, links and forms agents can use covers the fixes in detail, and alt text for AI agents covers product images.
\naria-label.<label>.div with a click handler looks like a button to a person and like plain text to the accessibility tree. Use <button> or <a href>.Accessibility work won't cover everything, and some of it doesn't matter to agents at all.
\nAgent readiness needs more than accessibility. A perfectly accessible site can still block AI assistants in robots.txt, challenge them with bot protection, or load its prices with JavaScript that many AI crawlers never run. Screen readers work inside a real browser, so they see JavaScript content; many agents don't. Structured data, sitemaps and llms.txt matter to agents and have little to do with accessibility. That's why AgentScore has Access and Readability categories as well as Navigability.
\nSome accessibility work matters less to agents. Color contrast, text resizing, captions and motion settings are essential for people and largely irrelevant to software. Don't use agent readiness as a reason to deprioritize them. They serve customers, and in many places they're a legal requirement.
\nBad ARIA hurts both. The W3C's own guidance is that no ARIA is better than bad ARIA. An aria-label that says \"button\", a role=\"button\" on something that can't be pressed, or aria-hidden=\"true\" on a real control misleads screen readers and agents alike. Native HTML elements are almost always the better choice.
A note for leaders: in the EU, the European Accessibility Act has applied to many e-commerce services since June 2025, so many retailers already have accessibility programs underway. If yours does, put agent readiness alongside it rather than starting a separate project. The same fixes, the same developers and often the same tickets.
\nAccessibility and agent readiness come from the same idea: a website should work for visitors who don't see it the way its designers do. More and more of those visitors are software acting for a person.
", "date_published": "2026-10-09T12:19:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Perspective" ] }, { "id": "https://ghostagentlab.com/blog/scanner-wears-other-names/", "url": "https://ghostagentlab.com/blog/scanner-wears-other-names/", "title": "Why our scanner sometimes wears other agents' names", "summary": "AgentScore requests a few pages as ChatGPT, Claude, Perplexity and Googlebot, unsigned, to see how your site treats them. Here's exactly what and why.", "content_html": "If you look closely at your logs during an AgentScore scan, you'll see a few requests that say they're ChatGPT, Claude, Perplexity or Googlebot. They aren't. They're our scanner, borrowing those agents' user agents to see how your site treats them. Here's why we do it, exactly what we send, and what the result can and can't tell you.
\nOne of the most common reasons an AI agent fails on a website has nothing to do with the website's design. The agent is turned away at the door. A bot protection rule, a firewall setting or a CDN feature decides that anything calling itself an AI agent gets a challenge page, an error, or a stripped-down page with no prices.
\nNobody notices, because people never see it. The site works perfectly in a browser. The only way to find out is to knock as the agent and compare what comes back with what a browser gets.
\nWe can't send the real ChatGPT to your site on demand. So for a handful of requests, our scanner sends the same User-Agent header those agents send, and compares the answer with a normal browser's.
| Name we use | What we request | Signed? |
|---|---|---|
AgentScore/1.0, our own name | robots.txt, sitemaps, llms.txt, well-known files (MCP and agentic commerce discovery), and checks for common pages such as /pricing and /cart | Yes |
| A normal Chrome browser | The home page, your llms.txt, and the key pages, before and after JavaScript runs. This is the baseline everything is compared with | No |
| ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot and Googlebot | The home page, once each | No |
| ChatGPT-User | The product, pricing and cart pages we found, once each | No |
| ChatGPT-User, Claude-User and Perplexity-User | The cart and checkout, once each | No |
The imitated requests use the user agent strings those operators publish, so they look the way the real thing does. We picked these agents because they're the ones that fetch pages for people in real time or power AI search, which are the visits most likely to turn into customers.
\nTraining crawlers such as GPTBot and ClaudeBot are handled differently. We read your robots.txt rules for them, along with every other AI agent we track, but we don't request pages as them. Blocking training is a business choice, and it doesn't cost points. Our guide to blocking or allowing AI crawlers covers that decision.
\nFor each imitated request, we line the response up against the browser's:
\nOn the home page, any blocked agent fails the Bot protection lets AI agents through check, and a degraded one is a warning. A similar comparison, looking for errors and challenge pages (and, at checkout, CAPTCHAs), runs on your product, pricing and cart pages for AI agents can open your product, pricing and cart pages, and on the cart and checkout for AI agents aren't blocked or shown a CAPTCHA at the cart and checkout. How AgentScore works explains how those checks add up.
\nEverything the scanner sends under its own name is signed with Web Bot Auth, so your CDN can prove it came from us. We explain how in Why we sign every request our agents send.
\nThe imitated requests are deliberately unsigned. If we signed them, a CDN that recognizes and trusts our signature might wave them through, and we'd be measuring how your site treats Ghost Agent Labs, not how it treats ChatGPT. Unsigned, they get exactly the treatment any request with that name gets.
\nOur imitation is honest about one thing it can't do: it can't pass identity checks. The requests come from our servers, not from OpenAI's, Anthropic's, Perplexity's or Google's, and they carry none of those companies' signatures.
\nSo if your bot protection verifies agents properly, checking published IP ranges or reverse DNS and blocking impostors, it may block our imitation too. That's the right behavior, and it will show up as a failed check. When that happens:
\nIf the real agents get through and only impostors are blocked, you're in good shape, whatever the check says. Our guides to telling whether an AI crawler is real and bot protection that lets AI agents through cover how to set that up.
\nThe opposite limit applies too. A site that treats these agents well when they come from our servers will probably treat the real ones well, but real products differ in details we can't copy, such as where their requests come from. Treat the result as a strong signal, not a guarantee.
\nImitated requests are a small part of a scan: a few page loads per agent at most, never a crawl. The scanner reads public pages only and never adds to a cart, submits a form or places an order. On the free AgentScore page, a domain's result is reused for 24 hours, so repeated requests for the same site don't cause repeated visits.
\nIf you see these requests in your logs: they'll arrive close together with a scan from AgentScore/1.0. The AgentScore bot page explains how to verify our signed requests and how to opt out, including from the comparison requests.
If you want to see the result: run AgentScore on your own site and open the Access checks.
\n\"Should we block AI?\" usually reaches a leadership meeting as one question. It's really three or four, because \"AI crawlers\" covers several kinds of software doing very different jobs. Block the wrong one and you vanish from the answers your customers are reading. Here's how to make the call, kind by kind.
\nThe major AI companies each run several agents, and each has its own name so you can treat them differently.
\n| Kind | What it does | Examples |
|---|---|---|
| Training crawlers | Collect pages to train future AI models | GPTBot, ClaudeBot, CCBot, Google-Extended (a control name, not a separate crawler) |
| AI search crawlers | Index pages so AI search can cite and link to you | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Assistant fetchers | Fetch a page in real time because a person asked about it | ChatGPT-User, Claude-User, Perplexity-User |
| Browser agents | Drive a real browser for a person: search, fill in forms, add to cart | Usually look like an ordinary browser |
Our robots.txt guide has the longer list and the exact names. The point for decision-makers is simpler: you can say yes to some and no to others.
\nEach visit stands in for a real person asking about you right now: \"Is this in stock?\", \"What's their returns policy?\", \"Which of these three is cheapest?\". Block them and the assistant answers without you, or recommends a competitor. For a store or a software company, this is the closest thing to a customer walking in.
\nThese build the index AI search draws on. They're the AI equivalent of being in Google's index, and they're how you get cited and linked. Few businesses that want search traffic would block Googlebot. The same logic applies here.
\nThis is the one worth debating. Allowing training means your content may shape what AI models know, including about your brand and products. Blocking it keeps your content out of future training sets, and leaves AI search and assistants unaffected, because they use different names.
\nEither answer is legitimate. That's why AgentScore reports blocked training crawlers but doesn't take points off for them.
\nBrowser agents mostly look like a normal browser, so robots.txt rules by name don't reach them. The useful question isn't whether to block them but whether they can complete a purchase or sign-up when they arrive. That's what Ghost Agent tests check.
\n| If you are... | Assistants and AI search | Training crawlers |
|---|---|---|
| An online store | Allow | Usually allow; your product pages are there to be read |
| A software or services company | Allow | Usually allow marketing and docs; your call on anything gated |
| A publisher or content business | Allow if you want AI citations and referral traffic | Often block, or allow only under a licensing agreement |
| Unsure | Allow | Decide deliberately; don't inherit someone's copied list |
One more nuance: some operators treat an assistant fetch, made because a person asked about a specific page, like a person clicking a link, and say robots.txt may not apply to it. If you truly need to stop those visits, that's a CDN rule, not a robots.txt line. Most businesses want those visits most of all.
\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Google-Extended\nDisallow: /\nUser-agent: *\nAllow: /\nSitemap: https://northwind.example/sitemap.xmlThe default for most businesses is simple: let the agents that bring customers in, make a deliberate choice about training, and enforce both where it actually counts.
", "date_published": "2026-10-09T12:17:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Perspective" ] }, { "id": "https://ghostagentlab.com/blog/90-day-agent-readiness-plan/", "url": "https://ghostagentlab.com/blog/90-day-agent-readiness-plan/", "title": "A 90-day agent readiness plan for e-commerce teams", "summary": "A practical three-phase plan: let AI agents in, make pages readable and usable, then prove agents can finish checkout, mapped to AgentScore checks.", "content_html": "Agent readiness can feel like a long list of unrelated fixes. It's easier to run as three phases of about a month each: let AI agents in, help them understand and move around your site, then prove they can finish the jobs that make you money. Here's a plan an e-commerce team can start on Monday.
\nName one person to own the plan. It doesn't have to be a developer; on most teams it's someone in e-commerce, digital or SEO who can get time from the people who make the changes. Our post on who owns agent readiness shows who usually fixes what.
\nThen agree what \"done\" means. We suggest three outcomes by day 90:
\nNotice that none of these is a target score. Scores are useful for tracking progress, but a site that loses ten points to an optional new standard can still serve agents well, and a site that scores well can still fail at the size selector. Aim at the outcomes.
\nThe first month is about Access, because nothing else matters if agents are turned away at the door. It's also when you set the baseline you'll measure against.
\n| AgentScore check | Usually fixed by | Read |
|---|---|---|
| robots.txt lets AI assistants and search agents in | SEO or content team | robots.txt for AI agents |
| Bot protection lets AI agents through | Bot protection or CDN admin | Tell if an AI crawler is real |
| No CAPTCHA or challenge on arrival | Bot protection or CDN admin | Bot protection and CAPTCHAs |
| AI agents can open your product, pricing and cart pages | Bot protection or CDN admin | Rate limits for AI agents |
| llms.txt guide for AI | SEO or content team | How to write an llms.txt file |
Make one policy decision this month too: what you'll do about AI training crawlers, as distinct from the assistants that fetch pages for shoppers. Our decision guide walks through it. Settle it early, because it affects robots.txt and bot protection rules at the same time.
\nEnd of month one: rescan. The AI assistants you want can reach your key pages, you've decided your position on training crawlers, and you have a baseline score and a first look at agent traffic.
\nMonth two covers Readability and Navigability: whether agents can understand your pages, and whether they can find their way around them. Most of this is developer work, so get it into the sprint plan early in the month.
\nEnd of month two: rescan and compare category by category with your baseline. Agents should be able to read the price, stock and description on your product pages from the HTML alone, and move around without getting stuck on unnamed controls or pop-ups.
\nMonth three is about Task completion, the category that carries the most weight. Checks can tell you a page looks usable. Only a real agent can tell you whether it is.
\nEnd of month three: at least one money journey runs on a schedule with alerts, failures go to a named owner, and leadership gets a one-page monthly view: AgentScore by category, Ghost Agent pass rates, agent traffic and AI referrals.
\nAgent readiness drifts. A new pop-up campaign, a bot protection rule change or a redesigned product page can undo months of work in one release. Keep three habits:
\nStart with a scan. It takes under a minute, and it'll tell you which month of this plan needs the most attention.
", "date_published": "2026-10-09T12:16:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Guide" ] }, { "id": "https://ghostagentlab.com/blog/who-owns-agent-readiness/", "url": "https://ghostagentlab.com/blog/who-owns-agent-readiness/", "title": "Who owns agent readiness?", "summary": "Agent readiness spans e-commerce, SEO, developers, security and legal. Who usually fixes what, and how to give the work one clear owner.", "content_html": "Ask who owns agent readiness and you'll often get a pause. SEO thinks it's a developer problem. Developers think it's a security setting. Security thinks it's marketing's call. Meanwhile AI agents are turned away, or get lost, and nobody hears about it. The work is spread across several teams by its nature. What it needs is one clear owner and a clear split of the rest.
\nAgent readiness touches four layers of a website, and each layer has its own team.
\nNo one team sees all four. Each sees its own layer working, and the failure shows up somewhere else: a lost recommendation, or an order that never happened.
\nIn the Ghost Agent Labs app, each AgentScore finding names a suggested owner under “Usually fixed by”, and the CSV export includes it too. Grouped together, the defaults look like this.
\n| Team | What they usually own | AgentScore checks |
|---|---|---|
| SEO or content team | The rules and text agents read first | robots.txt lets AI assistants and search agents in; llms.txt guide for AI; Clear page title, description, and headings; Images have text descriptions; Valid sitemap |
| Bot protection or CDN admin | Who gets through the door, and where | Bot protection lets AI agents through; No CAPTCHA or challenge on arrival; AI agents can open your product, pricing and cart pages; AI agents aren't blocked or shown a CAPTCHA at the cart and checkout |
| Developer | How pages are built and rendered | Content loads without JavaScript; Structured data describes your business and products; Product pages give price and stock in a form agents can read; Prices are in the page HTML; every Navigability check, from named buttons and labelled fields to pop-ups and checkout fields; Agents can use your site through MCP or WebMCP |
| E-commerce platform admin | How the store and checkout are configured | Shoppers can check out without an account; Agents can check out through an agentic commerce protocol |
Task completion, which comes from real agents running journeys, usually lands with developers once a test shows where an agent gets stuck. But the decision about which journeys matter belongs to the business.
\nThese are defaults, not rules. On a hosted platform, the \"developer\" fixes for a product page may really be theme settings an e-commerce manager can change. At a small company, one person may hold three of these roles.
\nThey own the outcome: whether AI agents can find, choose and buy from the site. That makes them the natural overall owner. They decide which journeys matter, set priorities when fixes compete for developer time, and own the platform settings that shape checkout, such as guest checkout. See guest checkout: why AI shopping agents need it.
\nThey own most of what agents read before they act: robots.txt, titles and headings, alt text, sitemaps and llms.txt. They also own measurement, because AI referrals belong next to search and social in channel reporting. See SEO got you found. Agent readiness gets you chosen and tracking visits and sales that come from AI assistants.
\nThey own the largest number of fixes: rendering content and prices in the HTML, structured data, named controls, labelled fields, dismissible pop-ups and checkout fields. Much of this overlaps with accessibility, so the same people and practices often apply. See accessibility work is agent readiness work. Developers also run agent tests as part of release checks, and evaluate newer interfaces such as MCP and WebMCP.
\nThey own the door. Bot protection, WAF rules, challenge pages and rate limits are where agents are most often blocked by accident, because those rules were written to stop scrapers. This team's job isn't to let everything in. It's to tell real agents from impostors, by verified identity rather than by name alone, and to treat the agents the business wants accordingly. See bot protection and CAPTCHAs, how to tell if an AI crawler is real and rate limits for AI agents.
\nLegal doesn't fix findings, but it shapes several decisions the other teams can't make alone:
\nA simple test: if an AI assistant started being blocked from your product pages tomorrow, who would find out, and how? If the answer is \"nobody\" or \"eventually\", that's the gap to close first.
\nFor a phased version of this, see our 90-day agent readiness plan. Or start by running AgentScore and seeing whose name comes up most.
", "date_published": "2026-10-09T12:15:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Perspective" ] }, { "id": "https://ghostagentlab.com/articles/mcp-webmcp-for-websites/", "url": "https://ghostagentlab.com/articles/mcp-webmcp-for-websites/", "title": "MCP and WebMCP: giving AI agents a direct way into your site", "summary": "What MCP servers and WebMCP tools are, what they mean for your business, how AgentScore detects them, and how to start safely.", "content_html": "Most AI agents use your website the way a person does: they read pages, click links and fill in forms. That works, but it's slow and it breaks easily. MCP and WebMCP give agents a second way in, a set of named actions such as \"search products\" or \"check order status\" that they can call directly. This guide explains both, what they mean for your business, and how to start small.
\nToday, an AI agent asked to find a waterproof jacket in a medium on your store has to work your pages. It loads the home page, finds the search box, reads the results, opens a product, works out the size picker, and reads the price. Every step depends on your page layout. A redesigned menu or a new pop-up can stop it.
\nThe alternative is to tell the agent what it can do and let it ask directly. Instead of driving the search box, it calls a search_products action with \"waterproof jacket, medium\" and gets back a clean list of products, prices and stock. That's the idea behind both MCP and WebMCP.
It doesn't replace a usable website. Many agents will keep using your pages for a long time, and the people they hand back to always will. Think of MCP and WebMCP as an extra, faster lane for the agents that support them.
\nThe Model Context Protocol (MCP) is an open protocol for connecting AI applications to outside tools and data. Anthropic introduced it in late 2024, and it has since been adopted by many AI assistants, developer tools and software companies.
\nAn MCP server is a small service that describes a set of tools (actions an agent can call, each with a name, a plain-language description and the inputs it expects) and can also offer resources (data it can read, such as a catalog or a policy). An AI application connects to the server, reads the list, and calls the tools it needs to complete a person's request.
\nFor a website, a remote MCP server runs alongside your site, usually on your own domain, and talks to the same systems your site does. Typical tools for an online store:
\n| Tool | What it does | Risk |
|---|---|---|
search_products | Finds products by keyword, category, size or price | Read-only |
get_product | Returns price, variants, stock and delivery estimate for one product | Read-only |
get_policy | Returns shipping, returns or warranty terms | Read-only |
get_order_status | Looks up an order for a signed-in customer | Needs sign-in |
add_to_cart | Builds a cart and returns a link for the person to check out | Changes state |
Software companies might offer tools to look up plans and pricing or search documentation. Service businesses might offer tools to check availability and request a booking.
\nCommerce platforms are starting to offer MCP servers for the stores they host, so check with yours before building one. If you have a developer team and an existing API, a read-only server is often a small project, because it wraps calls you already make.
\nWebMCP is an early proposal for bringing the same idea into the web page itself. Instead of running a separate server, your page registers tools with the browser through a new JavaScript API, navigator.modelContext. An agent working in that browser can then call the page's tools directly instead of clicking through it.
It's being developed in the open at the W3C's Web Machine Learning Community Group, with engineers from Google and Microsoft among those working on it. Community group work is incubation, not a finished standard, and browsers don't ship WebMCP by default yet. Expect the details to change.
\nTwo features make it interesting for websites:
\nA tool registered in a page looks roughly like this, guarded so it does nothing in browsers without the API:
\nif (\"modelContext\" in navigator) {\n navigator.modelContext.registerTool({\n name: \"search_products\",\n description: \"Search Northwind Outdoor products by keyword, size and maximum price.\",\n inputSchema: {\n type: \"object\",\n properties: {\n query: { type: \"string\" },\n size: { type: \"string\" },\n maxPrice: { type: \"number\" }\n },\n required: [\"query\"]\n },\n async execute({ query, size, maxPrice }) {\n const results = await searchCatalog({ query, size, maxPrice }); // your existing search\n return { content: [{ type: \"text\", text: JSON.stringify(results) }] };\n }\n });\n}The proposal also describes a declarative form: adding a toolname attribute (with a description) to an existing <form>, so a search or contact form becomes a tool without new JavaScript. Attribute names may change as the proposal develops.
| MCP server | WebMCP | |
|---|---|---|
| Where it runs | A service on your domain | Inside your web pages |
| Who can use it | AI applications that connect to MCP servers | Agents working in a browser that supports it |
| Maturity | Published specification, widely used | Early proposal, not shipped by default |
| Sign-in | Needs its own authorization for personal data | Uses the person's existing session |
| Best first step | Read-only catalog, pricing and policy tools | Mark up your search form, then experiment |
For most businesses, an MCP server is the practical choice today, and WebMCP is worth a small experiment so you're ready if browsers adopt it.
\nThe AgentScore check \"Agents can use your site through MCP or WebMCP\" is part of the Access category. It passes if it finds any of these:
\n/.well-known/mcp.json or /.well-known/mcp. There's no settled standard for advertising an MCP server yet, and proposals for a well-known server description are still being discussed, so we look in the most likely places.<link> or <a> whose rel or type mentions MCP.navigator.modelContext before your scripts run, so pages that check for the API register their tools as they would in a supporting browser. We record the tool names.toolname attribute.If none is found, the check is a warning rather than a failure. Agents can still use your pages; they just have to do it the slow way. It carries less weight than the checks that decide whether agents can get in at all, such as bot protection and robots.txt.
\nAdvertising a server isn't the same as it working. AgentScore confirms you've told agents where to find your tools. It doesn't call them. Test your tools with an MCP client before you publish the link, and keep them working as your catalog changes.
\n<link> in your page head.An MCP server or WebMCP tools won't bring traffic on their own. They make each agent visit more likely to end well: the right product found, the right price quoted, the cart built. Because support among AI applications is still growing, the best order of work is the usual one. Make sure agents can get in and read your pages first, then add a direct lane for the agents that can use it.
\nNot sure where you stand on the basics? Start with what agent readiness means, then run AgentScore to see every Access check, including this one.
", "date_published": "2026-10-09T12:14:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Access" ] }, { "id": "https://ghostagentlab.com/articles/agentic-commerce-protocols/", "url": "https://ghostagentlab.com/articles/agentic-commerce-protocols/", "title": "Agentic commerce protocols explained", "summary": "What ACP, UCP and AP2 are for, what they mean for your store, what to ask your platform, and what AgentScore can and can't detect.", "content_html": "Agentic commerce protocols let an AI assistant buy from your store on a person's behalf without filling in your checkout pages. The person says \"yes, buy it\" in the assistant, and the order arrives in your systems like any other. Several protocols have been announced in a short time, by different companies, and the names are easy to mix up. This guide explains what each one is for, what it means for your store, and what to ask your platform.
\nA browser agent can shop through your website, and for now that's how most agents buy. We cover that journey in how to make checkout work for AI shopping agents. But checkout pages were built for people. An agent filling them in has to guess at fields, cope with pop-ups and CAPTCHAs, and usually hand back to the person to pay.
\nA protocol replaces the page-filling with a structured conversation between the assistant and your store:
\nThis is a young and fast-moving area. The descriptions below are deliberately brief, and the protocols' own documentation is the place for detail. Programs, eligibility and regions change often.
\n| Protocol | Who's behind it | What it covers |
|---|---|---|
| Agentic Commerce Protocol (ACP) | OpenAI and Stripe | Checkout between an AI assistant and a merchant. It's the protocol behind buying inside ChatGPT, and it has been published as an open specification. |
| Universal Commerce Protocol (UCP) | Google, with industry partners | A broader standard for agents to discover what a store supports and complete checkout. A store publishes a profile at /.well-known/ucp. |
| Agent Payments Protocol (AP2) | Google, with payments industry partners | The payment step specifically: a way to prove that a person authorized an agent to make a purchase, designed to work with different payment methods and alongside other protocols. |
Card networks and payment providers have also announced their own programs for payments made by agents. Most of these sit underneath the protocols above, at the payment layer, rather than competing with them.
\nCheckout and payment are different jobs. ACP and UCP are about the shopping conversation and creating the order. AP2 is about trusting the payment. A store may end up supporting more than one, often without knowing, because the platform and payment provider handle it.
\nVery few stores will build a protocol integration themselves. Support usually arrives in one of three ways:
\nIf you run a custom stack, the specifications are public, but budget for real engineering: order creation, tax, shipping, inventory and payment all have to behave exactly as they do on your site.
\nFor stores, AgentScore includes the Access check \"Agents can check out through an agentic commerce protocol\". It's skipped for sites that aren't stores.
\nIt can only see what a store publishes openly. Today that means a Universal Commerce Protocol profile: the scanner requests /.well-known/ucp, and if it gets back a JSON document, the check passes.
The Agentic Commerce Protocol is different. Integrations are set up privately between each store and each AI platform, so there's nothing on your site for a scanner to find. If you support ACP but not UCP, this check will still show a warning. That's a limit of what can be seen from outside, not a problem with your store. The advice in the report names both protocols.
\nA warning here costs little. The check has a small weight in the Access category, because most agents still shop through the website, and that's where the checks that decide whether agents can get in and get through checkout matter more. Those include bot protection, CAPTCHAs at the cart and checkout, and guest checkout.
\nAgentic commerce protocols will keep changing for a while. The stores best placed to benefit are the ones whose websites, data and policies already work for agents, because that's what every protocol is built on.
", "date_published": "2026-10-09T12:13:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Access" ] }, { "id": "https://ghostagentlab.com/articles/guest-checkout-ai-agents/", "url": "https://ghostagentlab.com/articles/guest-checkout-ai-agents/", "title": "Guest checkout: why AI shopping agents need it", "summary": "Why forced accounts stop AI agents at the last step, how AgentScore checks for guest checkout, and how to offer it without losing accounts.", "content_html": "An AI agent can find your product, choose the size and fill the cart, then stop dead at a page that says \"Sign in to continue\". It can't create an account for the person it's shopping for, and it usually can't sign in as them either. Guest checkout removes that dead end. This guide explains why it matters, how AgentScore checks for it, and how to offer it without giving up on customer accounts.
\nA forced account is a known source of abandoned checkouts with people. With AI agents it's usually worse, for practical reasons:
\nGuest checkout fixes all four. The agent, or the person it hands back to, enters an email address and a delivery address and carries on to payment.
\nautocomplete value. See buttons, links and forms agents can use.Accounts matter to most stores: repeat purchases, order history, loyalty and marketing consent. Guest checkout doesn't mean giving those up. It means moving the ask to a moment when it doesn't cost you the sale.
\n| Instead of | Try |
|---|---|
| Requiring an account before checkout | Offering \"Save your details for next time\" on the order confirmation page |
| Asking for a password at checkout | Emailing a link to set up an account from the order |
| Hiding order tracking behind sign-in | Order tracking by order number and email address |
| Requiring sign-in for loyalty points | Matching guest orders to existing accounts by email, then awarding points |
| An account wall for members-only prices | Showing the public price to guests and the member price once signed in |
Some stores genuinely need accounts: trade-only wholesale, age-restricted products, prescription items, or subscription services where the account is the product. If that's you, say so clearly on product pages and in your llms.txt, so agents can tell the person up front rather than discovering it at checkout.
\nThe Access check \"Shoppers can check out without an account\" applies to stores. The scanner is read-only: it loads your cart and checkout pages, but never submits a form or adds anything to a cart. It looks at what a shopper would see:
\nIt carries a moderate weight in the Access category, alongside the related check that AI agents aren't blocked or shown a CAPTCHA at the cart and checkout. If your checkout couldn't be tested, a Ghost Agent can walk the full journey with items in the cart. See how to test your key journeys with AI agents.
\nPlatform settings often decide this. Most commerce platforms have a single setting for whether accounts are required, optional or disabled at checkout. If yours is set to required, changing it may be the quickest win on your whole AgentScore report. Check with your platform admin before redesigning anything.
\nAssistants that buy through an agentic commerce protocol don't use your checkout pages, so they bypass this problem. But most agents still shop through the website, and the people they hand back to always do. Guest checkout is the simplest change that makes both work, and it helps human shoppers in a hurry just as much.
\nFor the rest of the journey, from search to the payment handoff, see how to make checkout work for AI shopping agents. Then run AgentScore to see where your store stands.
", "date_published": "2026-10-09T12:12:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Access" ] }, { "id": "https://ghostagentlab.com/articles/rate-limits-ai-agents/", "url": "https://ghostagentlab.com/articles/rate-limits-ai-agents/", "title": "Rate limits for AI agents: fair limits that don't turn customers away", "summary": "How to set rate limits that stop scrapers but not AI assistants: 429 with Retry-After, per-agent limits for verified agents, and what to watch.", "content_html": "Rate limits protect your site from scrapers and floods of automated traffic. Set badly, they also turn away the AI assistants that are looking up your products for a customer. This guide explains how to set limits that stop abuse without blocking the agents you want, and how to say \"slow down\" in a way agents understand.
\nMost rate limits count requests per IP address. That made sense when one address meant one visitor. It works less well for AI agents:
\nHTTP has a status code made for this. 429 Too Many Requests, defined in RFC 6585, tells the client it has sent too many requests. Pair it with a Retry-After header, defined in RFC 9110, saying how many seconds to wait:
\nHTTP/1.1 429 Too Many Requests\nRetry-After: 30\nContent-Type: text/plain\nToo many requests from this client. Please wait 30 seconds and try again.Well-behaved crawlers and agents treat this as an instruction, not a door slammed shut. Google's crawler documentation, for example, says Googlebot slows down when it gets 429, 500 or 503 responses. Compare the alternatives:
\n| Response | What the agent concludes |
|---|---|
| 429 with Retry-After | Wait, then try again. The site is fine. |
| 429 without Retry-After | Back off, but guess for how long. |
| 503 Service Unavailable | The site is having problems. Fine for real outages, misleading for rate limits. |
| 403 Forbidden | I'm not allowed here. Many agents give up and say so. |
| 200 with a challenge or \"access denied\" page | Confusing. Some agents read the challenge text as your content. |
Keep the 429 lightweight: a short plain-text message, no heavy page, and never a CAPTCHA, which agents can't solve. There's also an IETF draft for RateLimit headers that tell clients their remaining allowance before they hit the limit. It's still a draft, so treat it as a bonus for API clients, not a replacement for Retry-After.
The fairest limits are set per agent, based on verified identity. Most CDNs and bot management products can verify the major AI agents against their operators' published IP ranges, reverse DNS or signed requests (see how to tell if an AI crawler is real). Once you know who's asking, you can give each kind of visitor a limit that fits its job:
\n| Visitor | What it's doing | Suggested approach |
|---|---|---|
| Verified AI assistants (such as ChatGPT-User, Claude-User, Perplexity-User) | Fetching pages because a person asked | Generous limits that allow short bursts. Each request stands in for a customer. |
| Verified AI search crawlers (such as OAI-SearchBot) | Indexing pages so you appear in AI answers | A steady crawl rate. Use 429 with Retry-After to pace them, not blocks. |
| Verified AI training crawlers (such as GPTBot) | Collecting content for model training | Your business decision. Pace them, or opt out in robots.txt. See robots.txt for AI agents. |
| Unverified traffic claiming an AI agent's name | Unknown, sometimes impersonation | Normal or stricter limits. Never give extra allowance to a name alone. |
| Everything else | People, browsers, other bots | Your existing limits, tuned so a busy shopper never hits them. |
The right numbers depend on your traffic and infrastructure, so we don't suggest specific figures. Start from what your logs show a normal agent visit looks like, then set limits comfortably above it.
\nAbuse mostly targets actions: logins, account sign-ups, discount codes, gift card balances, stock checks and search. Pages that only display information are cheap to serve, especially from a cache. Put your tightest limits on:
\nKeep product, category, pricing, policy and help pages loose, and cache them well. Those are the pages agents need to answer questions about you, and the ones that turn into sales.
\nETag and Last-Modified headers let crawlers ask \"has this changed?\" and get a tiny 304 Not Modified response if not.lastmod dates help crawlers revisit only what changed. See XML sitemaps for AI agents.A quick checklist. Rate-limited responses use 429 with Retry-After, never 403 or a CAPTCHA. Verified AI assistants have their own, more generous limits. Names alone earn nothing extra. The strictest limits sit on logins, coupons and search, not on product and policy pages. And someone checks the 429 rate by agent at least monthly.
\nRate limits and bot protection work together. For the wider picture, read bot protection and CAPTCHAs and, for the business decision about which agents to let in at all, should you block AI crawlers?
", "date_published": "2026-10-09T12:11:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Access" ] }, { "id": "https://ghostagentlab.com/articles/image-alt-text-ai-agents/", "url": "https://ghostagentlab.com/articles/image-alt-text-ai-agents/", "title": "Alt text for AI agents: describing product images", "summary": "Why AI agents rely on alt text, what AgentScore checks, and how to write useful descriptions for product photos across a whole catalog.", "content_html": "Your product photos do a lot of selling. They show the color, the cut, the size in someone's hand. Most AI agents never see them. What they read instead is the alt text: the short description attached to each image in your HTML. If it's missing, the photo might as well not be there.
\nAn AI agent helping someone shop usually reads your page as text. It fetches the HTML, strips out the layout and works from what's left. Images arrive as file names and URLs, which tell it very little. IMG_4471_final.jpg doesn't say whether the jacket is navy or black.
Some agents do look at the page as a picture, taking screenshots and reading them with a vision model. They can make out a photo, but slowly, at a cost, and not always correctly. Text is faster, cheaper and unambiguous, so even those agents lean on it when it's there.
\nAlt text is the bridge. It's an attribute on the image tag, written for anyone who can't see the image: screen reader users, people on slow connections, search engines and now AI agents. Writing it well is the same work for all of them. We make that case in Accessibility work is agent readiness work.
\nThe \"Images have text descriptions\" check is part of the Readability category. It loads your home page in a real browser and looks at the images people can actually see, ignoring tiny ones under 32 pixels wide such as icons and tracking pixels. For each one it asks a simple question: does it have an alt attribute at all?
| Result | What it means |
|---|---|
| Pass | 90% or more of visible images have alt text, or there are no content images on the page |
| Warning | Between 60% and 90% have alt text |
| Fail | Fewer than 60% have alt text |
An empty alt=\"\" counts as described. That's deliberate: it's the correct way to mark an image as decorative, and it tells agents and screen readers to skip it. The check can't judge whether your descriptions are any good, so that part is up to you.
It's a small check by weight, because a missing description rarely stops an agent on its own. But the same gaps tend to show up elsewhere. A linked product image with no alt text is also a link with no name, and that's caught by the \"Buttons and links have names\" check in Navigability, which carries more weight.
\nDescribe what a shopper needs to know from the photo, in the words they'd use. Lead with the product, then add the details the image actually shows.
\n| Weak | Better |
|---|---|
alt=\"product\" | alt=\"Northwind waxed cotton field jacket in navy, front view\" |
alt=\"IMG_4471\" | alt=\"Field jacket in navy, close-up of brass zip and corduroy collar\" |
alt=\"best jacket waterproof jacket mens jacket sale\" | alt=\"Model, 6 ft 1 in, wearing the field jacket in size L\" |
| No alt attribute | alt=\"\" if the image is purely decorative |
Not every image needs words. Background textures, divider graphics and lifestyle photos that repeat the headline next to them can take alt=\"\". That's different from leaving alt out: an empty value says \"nothing to see here\", while a missing one leaves agents and screen readers guessing, and some will read out the file name instead.
Images that are links need extra care, because the alt text becomes the link's name. A product tile where the whole image links to the product page should have alt text that works as a link, such as the product name. Your logo in the header links home, so alt=\"Northwind home\" or simply your brand name works better than alt=\"logo\".
Text baked into images is a common trap on home pages. A banner that says \"40% off outerwear this weekend\" as pixels is invisible to most agents. Put the offer in real HTML text over the image, or at minimum repeat it in the alt text.
\nA store with thousands of products can't write every description by hand. You don't need to. Most of it can be generated from data you already have.
\nalt attribute on every product image, and falls back to the product name and variant when no custom text is set. That alone gets most stores to a pass.Check it yourself in a minute: open a product page, right-click a photo and choose Inspect. Look for alt=\"…\" on the <img> tag. If it's missing, or it just says \"image\", agents are getting nothing.
Alt text is the visual part of a bigger job: making sure everything a shopper learns from your page is also available as text. The other parts are prices and stock in a form agents can read (see Prices AI agents can read and Product data that AI shopping agents can read), and content that's in the HTML rather than added by JavaScript (see our guide to JavaScript-only content).
\nRun AgentScore to see how your home page does, then spot-check a few product pages by hand. For the full picture of what agents need from a site, start with the agent readiness guide.
", "date_published": "2026-10-09T12:10:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/", "url": "https://ghostagentlab.com/articles/xml-sitemaps-ai-agents/", "title": "XML sitemaps for AI agents", "summary": "How AI agents and crawlers use your sitemap, what AgentScore checks, what to include and how to keep it accurate.", "content_html": "A sitemap is a list of the pages on your site that you want found, in a format machines read without effort. Search engines have used them for about twenty years. AI agents and AI search crawlers use them too, to find pages your navigation hides and to discover new products quickly. It's one of the cheapest readiness wins there is.
\nAn agent arriving at your site has two ways to find things: follow links from the home page, or read a list someone has prepared. Following links is slow and fragile. Menus built with JavaScript, products buried four clicks deep and pages that only appear in search results can all be missed.
\nA sitemap solves that. It's an XML file, usually at /sitemap.xml, that lists your important URLs. Crawlers from AI search engines use it to decide what to fetch. Agents doing a task can use it to locate a product or policy page directly. And it gives every system the same, complete picture of what you publish.
It doesn't replace good links. A page that's only in your sitemap is still a page most visitors and many agents never reach. Treat the sitemap as the safety net under your navigation, not instead of it. Our guide to linking key pages from the home page covers the other half.
\nThe \"Valid sitemap\" check, in the Readability category, looks for your sitemap the way crawlers do:
\nSitemap: lines in your robots.txt./sitemap.xml and /sitemap_index.xml.<urlset> or <sitemapindex> element.It passes when it finds one, and reports how many entries it holds and whether it's listed in robots.txt. It fails if none of those places has a valid sitemap. A home page or a \"not found\" page served at /sitemap.xml doesn't count, which catches a common problem with some site builders.
The sitemap also feeds other checks. AgentScore uses it to find a product or pricing page when the home page doesn't link to one. If the sitemap is an index, it reads product sitemaps first, since most platforms name them that way. That's why a missing sitemap can leave other checks marked \"not tested\": without a product page to look at, there's nothing to check prices or product data on.
\nThe format is defined at sitemaps.org. A minimal one for a small store:
\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n <url>\n <loc>https://northwind.example/</loc>\n <lastmod>2026-10-01</lastmod>\n </url>\n <url>\n <loc>https://northwind.example/products/ethiopia-yirgacheffe</loc>\n <lastmod>2026-10-07</lastmod>\n </url>\n <url>\n <loc>https://northwind.example/pages/shipping</loc>\n <lastmod>2026-08-14</lastmod>\n </url>\n</urlset>Larger sites split it up. Each sitemap file can hold up to 50,000 URLs, and a sitemap index lists the individual files:
\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<sitemapindex xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n <sitemap><loc>https://northwind.example/sitemap_products_1.xml</loc></sitemap>\n <sitemap><loc>https://northwind.example/sitemap_collections_1.xml</loc></sitemap>\n <sitemap><loc>https://northwind.example/sitemap_pages_1.xml</loc></sitemap>\n</sitemapindex>Then tell crawlers where it is, with one line in robots.txt:
\nSitemap: https://northwind.example/sitemap.xml| Include | Leave out |
|---|---|
| Product and category pages you want people to find | Cart, checkout and account pages |
| Pricing, plans and sign-up pages | Internal search results and filtered URLs |
| Shipping, returns, FAQ and contact pages | Pages that redirect elsewhere |
| Guides, articles and other evergreen content | Pages marked noindex, or blocked in robots.txt |
| The canonical URL of each page | Duplicates with tracking parameters or session IDs |
The rule of thumb: list the pages you'd be happy for an AI assistant to send a customer to, each at its one true address. If you include a page in the sitemap but block it in robots.txt, you're sending crawlers mixed signals.
\nlastmod honest. It should change when the content does, such as a price or stock change, not every time the file is rebuilt. Google's documentation says it uses lastmod only when it's consistently accurate, and other crawlers are likely to treat it similarly.Both, because they do different jobs. A sitemap is complete: every page you want found, with no explanation. An llms.txt file is selective: a short, described guide to the pages that matter most, written for AI. A crawler indexing your catalog wants the sitemap. An assistant trying to answer \"what's their returns window?\" benefits from llms.txt pointing straight at the returns page.
\nQuick check: open yourdomain.com/robots.txt and look for a Sitemap: line. Then open the URL it gives. You should see XML listing your pages, not a web page. If either step fails, that's your first fix.
Sitemap: line to robots.txt if it's missing.For how the sitemap fits with the other things agents need, see the agent readiness guide.
", "date_published": "2026-10-09T12:09:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/titles-descriptions-headings-ai/", "url": "https://ghostagentlab.com/articles/titles-descriptions-headings-ai/", "title": "Titles, descriptions and headings AI agents understand", "summary": "How AI agents use page titles, meta descriptions, headings and the lang attribute, and how to write each so agents pick the right page.", "content_html": "Before an AI agent reads a page in full, it reads the labels: the title, the description and the headings. They're how it decides whether a page answers the question it was asked, and how it finds the right part of a long page. They're also the parts of a page most often left on autopilot.
\nA person landing on your page takes in the design at a glance: the hero image, the logo, the layout. An agent sees none of that. It sees text, in order, and leans heavily on the few pieces of text that say what the page is.
\n<title> is the page's name. It shows in browser tabs and search results, and it's often the first thing an agent records about a page it has visited.<h1> to <h6>) are the outline. An agent looking for \"delivery times\" on a long FAQ uses them to jump to the right section.lang attribute on the <html> tag says what language the page is in, so the agent reads and quotes it correctly.If you've done SEO work, most of this will be familiar. The difference is that an agent isn't ranking your page among ten others; it's trying to complete a task, and a vague label can mean it picks the wrong page or gives up. There's more on that shift in SEO got you found. Agent readiness gets you chosen.
\nThe \"Clear page title, description, and headings\" check, in the Readability category, reads your home page's HTML before any JavaScript runs and looks for four things:
\n| Looks for | Counts as missing when |
|---|---|
A <title> | There's no title tag, or it's empty |
| A meta description | There's no <meta name=\"description\">, or it's empty |
An <h1> heading | There's no h1 in the HTML (one added later by JavaScript doesn't count) |
A lang attribute | The <html> tag has no lang |
All four present is a pass, and the result shows your title. One or two missing is a warning. It fails if there's no title at all, or if three or more are missing. The check confirms these things exist; it can't tell whether the wording is helpful. The rest of this guide is about that part.
\nA good title says what the page is and whose it is, specific enough to tell it apart from every other page on your site.
\n| Page | Weak | Better |
|---|---|---|
| Home page | Home | Northwind Coffee: specialty coffee beans and brewing gear |
| Product | Shop | Ethiopia Yirgacheffe whole bean coffee, 340 g | Northwind Coffee |
| Category | Collection | Pour-over coffee kits | Northwind Coffee |
| Policy | Info | Shipping rates and delivery times | Northwind Coffee |
Write the meta description as the answer to \"what will I find here?\" One or two plain sentences: what the page offers, for whom, and anything that sets it apart, such as price range, delivery area or a free trial.
\n<meta name=\"description\" content=\"Light-roast single-origin coffee from\nYirgacheffe, Ethiopia, with notes of jasmine and lemon. Roasted to order,\n340 g bag, ships free in the US over $40.\">Avoid repeating the title, stacking keywords, or reusing one description across many pages. If a platform fills it in automatically from the first lines of the page, check that those lines are useful. Cookie notices and \"Free shipping this week\" banners make poor summaries.
\nHeadings should read like a table of contents. Someone reading only your headings should understand what the page covers and where each part is.
\nA product page outline might look like this:
\nh1 Ethiopia Yirgacheffe whole bean coffee\n h2 Tasting notes\n h2 Brewing suggestions\n h2 Shipping and returns\n h2 ReviewsIt's one attribute, and it's often missing from custom themes: <html lang=\"en\">. Use the right code for each language version, such as lang=\"fr\" or lang=\"en-GB\". It helps agents interpret the page, translate it correctly and quote it in the right language. Screen readers use it to choose a voice.
Who owns this: usually the SEO or content team, with a developer for template changes and the lang attribute. It's rarely a big project, which is why it's worth doing first.
\nClear labels are one part of a readable site. For the rest, see the agent readiness guide.
", "date_published": "2026-10-09T12:08:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/machine-readable-prices/", "url": "https://ghostagentlab.com/articles/machine-readable-prices/", "title": "Prices AI agents can read", "summary": "How to show prices AI agents read correctly: in the server HTML, with a clear currency, labelled sale prices, variant prices and one source of truth.", "content_html": "\"How much is it?\" is one of the first things a shopper asks an AI assistant about a product. If an agent can't find your price, or finds two prices and can't tell which is real, it either guesses or moves on to a store where the answer is clear. Here's how to make your prices unmistakable.
\nAgents find prices in two places. The first is the text of the page: the \"$18.00\" a shopper sees next to the add-to-cart button. The second is structured data: a block of schema.org JSON-LD in the HTML that states the price, currency and stock as plain values. Good stores have both, and they agree.
\nBoth have to be in the HTML your server sends. Many agents, and most AI search crawlers, read that HTML without running JavaScript. If your theme loads prices from an API after the page appears, those agents see a product with no price. To a shopper asking an assistant to compare three coffee grinders, that's a product that doesn't make the list.
\nTwo Readability checks cover prices.
\n| Check | What it looks at |
|---|---|
| Prices are in the page HTML | Your pricing page, or a product page if there's no pricing page. It reads the text of the HTML before JavaScript runs and looks for something that's clearly a price: an amount with a currency symbol ($, €, £, ¥, ₹) or a currency code such as USD, EUR, GBP, CAD or AUD. It passes if it finds one. It warns if prices only appear after JavaScript runs in a real browser, or if it can't find a price at all. |
| Product pages give price and stock in a form agents can read | A product page's HTML, for schema.org Product or ProductGroup data with an offer that gives a price, a currency and availability. It passes with all three, warns if any are missing, if there's only microdata, or if the data is only added by JavaScript, and fails if there's none. Stores only. |
AgentScore finds these pages from links on your home page, then from your sitemap, then a common address such as /pricing. If it can't find one, the checks are marked \"not tested\" rather than failed.
This guide focuses on the visible price and keeping everything consistent. For a full walk-through of product structured data, see Product data that AI shopping agents can read.
\nOpen a product page, right-click and choose View Page Source (not Inspect, which shows the page after JavaScript). Search for the price. If it's there, agents can read it. If it isn't, the fix belongs to your developer or platform: render the price on the server, as part of the page, and let JavaScript update it afterwards if a shopper changes options.
\n<p class=\"price\">\n <span>$18.00</span> <span>USD</span>\n</p>For software and services, the same applies to your pricing page: each plan's price, billing period and what it includes should be readable text, not an image or a widget that loads later. If you don't publish prices, say so in plain words, for example \"Pricing is quoted per project. Contact sales for a quote.\" Then an agent can tell people how to get one, instead of reporting that it couldn't find any.
\n\"$18\" means different things in the US, Canada and Australia. A person infers the currency from where they are. An agent may be working for someone elsewhere, fetching your page from a data center in another country.
\npriceCurrency in structured data using the three-letter ISO 4217 code, such as USD or EUR.Discounts are where agents most often misread prices. A page showing \"$24.00 $18.00\" with the first one struck through looks obvious to a person, because they can see the line through it. An agent reading text may see two prices side by side.
\n<del> or <s> for the original price, not just a CSS line-through.price should be the current sale price. If the sale ends on a known date, priceValidUntil says so.<p class=\"price\">\n Was <del>$24.00</del>, now <strong>$18.00 USD</strong>\n <span>Sale ends October 31</span>\n</p>When sizes, colors or pack sizes have different prices, agents need to know which price goes with which option. A single price that changes with JavaScript when a shopper picks a size is invisible to an agent that doesn't click.
\nProductGroup with hasVariant gives each option its own offer with its own price and availability. AgentScore reads offers inside variants too.An agent may see your price in several places: the page text, the JSON-LD, your product feed in Google Merchant Center or other shopping channels, and the cart. If they disagree, the agent can't know which is right. It may quote the wrong one, or treat your store as unreliable. Search engines may ignore structured data that doesn't match the page.
\nOne source of truth: generate the visible price, the structured data and your product feeds from the same data, so a price change updates all of them together. Hand-entered prices in a separate feed or a hard-coded JSON-LD block are the usual cause of mismatches.
\nUnit prices help too. For groceries, coffee, cosmetics and anything sold by weight or volume, a unit price such as \"$5.29 per 100 g\" lets an agent compare fairly across pack sizes.
\nPrices are one of the facts agents most need to get right before they recommend you or buy from you. For the step after that, getting the item into a basket and through checkout, see our guide to agent-ready checkout.
", "date_published": "2026-10-09T12:07:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/policy-pages-ai-assistants/", "url": "https://ghostagentlab.com/articles/policy-pages-ai-assistants/", "title": "Shipping, returns and FAQ pages AI assistants can quote", "summary": "How to write shipping, returns and FAQ pages that AI assistants can find, read and quote accurately to shoppers before they buy.", "content_html": "\"Do they ship to Canada?\" \"Can I return it if it doesn't fit?\" \"How long does delivery take?\" These are the questions people ask AI assistants just before they buy. The answer comes from your shipping, returns and FAQ pages, quoted word for word or summarized. If those pages are vague, hidden or contradictory, the assistant gets it wrong, or tells the shopper it isn't sure.
\nPolicy pages used to be read by a small share of careful shoppers, and by your support team. Now an assistant may read them on behalf of every shopper who asks a question about delivery or returns. Whatever it says becomes your answer, with your name on it.
\nThat has two consequences. A clear policy page means accurate answers and confident shoppers. A muddled one means an assistant may misquote your returns window, invent a shipping rate, or decline to answer and suggest a store whose terms it could read. Neither shows up in your analytics as an error. It shows up as sales that didn't happen, or customers who expected something you never promised.
\nAgentScore doesn't have a dedicated check for policy pages yet, so this guide is about what you can do and test yourself.
\nAssistants often lift a sentence or two out of a page. Write so that any sentence still makes sense when it's quoted without its neighbors.
\n| Hard to quote | Easy to quote |
|---|---|
| We're happy to help with returns within the usual period, subject to the conditions above. | You can return unopened items within 30 days of delivery for a full refund. |
| Shipping is fast and affordable. | US orders ship in 1 to 2 business days. Standard delivery costs $5.95 and takes 3 to 5 business days. Orders over $40 ship free. |
| International shipping may be available. | We ship to the US and Canada. We don't ship to other countries yet. |
| See exceptions. | Opened coffee and gift cards can't be returned. |
Organize each page by the questions customers actually ask. Your support inbox and chat logs are the best source. Use those questions, or close versions of them, as headings, and answer directly underneath. That gives an agent a clear outline to search, and the heading itself matches the question it was asked. Our guide to titles, descriptions and headings covers how to build that outline.
\nA shipping page might look like this:
\nh1 Shipping and delivery\n h2 Where do you ship?\n h2 How much does shipping cost?\n h2 How long does delivery take?\n h2 Can I track my order?\n h2 What if my order arrives damaged?Use plain HTML tables for anything with more than two variables, such as rates by region and speed. A table reads clearly as text; a rate chart saved as an image doesn't read at all.
\nContradictions are the fastest way to an inaccurate answer. If the FAQ says 30-day returns, the returns page says 14 days and a product page says \"free returns for 60 days\", an assistant has to pick one, and it may not pick the one you meant.
\nIf your policies vary by country, membership or product type, say so clearly and early. One page per region, or a clearly labelled section for each, works better than footnotes. Name the region in the heading: \"Returns from Canada\", not \"Other regions\". If marketplace sellers on your site set their own policies, say that too, and say where the shopper can find them.
\nTest it the way your customers will: ask two or three AI assistants the questions your support team hears most, naming your store. \"What's Northwind Coffee's return policy?\" \"Does Northwind ship to Canada, and how much does it cost?\" Compare the answers with your actual policy. Where they're wrong or hesitant, the page that should have answered is the one to rewrite.
\nPolicy pages are where an assistant's answer turns into a sale or a lost one. Once they're clear, run AgentScore to check the rest of your site, and read the agent readiness guide for the bigger picture.
", "date_published": "2026-10-09T12:06:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/", "url": "https://ghostagentlab.com/articles/cookie-banners-popups-ai-agents/", "title": "Cookie banners and pop-ups that trap AI agents", "summary": "How cookie banners, sign-up pop-ups and chat widgets block AI agents, what AgentScore checks, and how to design overlays agents can dismiss.", "content_html": "The first thing many AI agents see on a website isn't the home page. It's a cookie banner, a discount pop-up or a country selector sitting on top of it. A person closes it without thinking. An agent has to find the right button, work out what it means, and press it, and if it can't, the visit ends there. This guide covers the overlays that cause trouble, what AgentScore checks, and how to design them so they stay out of the way.
\nBrowser agents, the kind that open a real browser and click through a site for someone, read a page through its structure and often a screenshot as well. An overlay that covers the page changes both. In the screenshot, the content is hidden or dimmed. In the page structure, a modal can make everything behind it unreachable until it's closed.
\nSo the agent's first job is to get rid of it. That goes wrong in a few predictable ways:
\n<div> or <span> that responds to a mouse click but isn't a real button. Many agents won't recognize it as something they can press.For the business, this is a quiet loss. The agent doesn't complain or fill in a feedback form. It tells the person it couldn't get through, or moves on to a site where it could.
\nThe AgentScore check is called \"Pop-ups and banners can be dismissed by agents\", and it's one of the higher-weighted checks in the Navigability category. It works like this:
\n| What AgentScore finds | Result |
|---|---|
| Nothing covers the page on arrival | Pass |
| An overlay covers the page, but it has a clearly labelled button to close or answer it | Warning: agents can get past it, but every one has to deal with it first |
| An overlay covers the page and has no clearly labelled way out | Fail |
A second check, \"Agents can find and press your add-to-cart or sign-up button\", catches the other half of the problem. It finds the main button on a product or pricing page and tests whether something else, usually a pop-up, banner or chat widget, sits on top of it. If it does, the check fails, because an agent's click would land on the overlay instead.
\nThis isn't about skipping consent. Where the law requires you to ask for cookie consent, keep asking. The goal is a banner that an AI agent acting for a person can answer as easily as the person could, not one that disappears. Talk to whoever owns privacy before changing what your banner asks or when.
\n| Overlay | What tends to go wrong | Better approach |
|---|---|---|
| Cookie consent banner | Full-screen modal; icon-only close; reject hidden in settings | A bar at the bottom of the screen with \"Accept all\" and \"Reject all\" as real, named buttons side by side |
| Newsletter or discount pop-up | Appears on arrival, covers the page, close button is a small unlabelled icon | Wait until someone has engaged; use a labelled \"Close\" or \"No thanks\"; don't show it on product or checkout pages |
| Country or region selector | Blocks the page until a choice is made, often with a custom dropdown | A small, dismissible suggestion (\"Shopping from Canada? Go to the Canadian store\") and locale in the URL |
| Age check | Custom date pickers or a fake button | Where required, a short form with a native field and a real \"Confirm\" button |
| Chat widget | Opens itself on load, or its launcher covers buttons on smaller screens | Start collapsed, keep the launcher small and clear of the main buttons |
| App install or promo banner | Sticky and tall, pushing content down or covering it | Keep it short, dismissible with a named button, and remember the choice |
Here's a cookie banner that people, screen readers and AI agents can all handle. It sits at the bottom of the page, doesn't block what's behind it, and every choice is a real button with a plain name.
\nTraps agents
\nWorks for agents
\n<div class=\"consent-bar\" role=\"region\" aria-label=\"Cookie choices\">\n <p>We use cookies to run the store and, with your OK, to measure ads.\n <a href=\"/pages/cookies\">Cookie policy</a></p>\n <button type=\"button\">Reject all</button>\n <button type=\"button\">Accept all</button>\n <button type=\"button\">Choose cookies</button>\n</div>And the most common failure, the icon-only close button, fixed in one attribute:
\n<!-- An agent sees: button, no name -->\n<button class=\"modal-x\"><svg>…</svg></button>\n<!-- An agent sees: button \"Close\" -->\n<button class=\"modal-x\" aria-label=\"Close\"><svg aria-hidden=\"true\">…</svg></button>A few rules that make the difference:
\naria-label.The home page is where AgentScore looks for overlays, but the pages that matter most are product pages, the cart and checkout. That's where a covered button costs a sale. Check those pages on a narrower screen too: a chat launcher that sits harmlessly in a corner on a laptop can land on top of \"Add to cart\" on a phone-sized window.
\nIf marketing relies on an email capture pop-up, that's a reasonable trade-off to discuss rather than a rule to break. Showing it after someone has scrolled or spent time on the page, rather than on arrival, keeps most of its value for people and removes it from the first moment an agent arrives.
\nOverlays are one part of how agents find their way around a page. For the rest, see buttons, links and forms agents can use, and for the other thing that often greets agents on arrival, a CAPTCHA or bot challenge, see bot protection and CAPTCHAs. To see how agents handle the whole journey, not just the first screen, read how to test your key journeys with AI agents.
", "date_published": "2026-10-09T12:05:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Navigability" ] }, { "id": "https://ghostagentlab.com/articles/link-key-pages-home-page/", "url": "https://ghostagentlab.com/articles/link-key-pages-home-page/", "title": "Link your key pages from the home page", "summary": "AI agents start at your home page and follow links. Make sure products, the cart and pricing are linked in plain HTML so every agent can find them.", "content_html": "When an AI agent is asked about your business, it usually starts where people do: your home page. From there it follows links. If the links to your products, cart or pricing aren't in the page's HTML, many agents never find those pages, however good they are. It's one of the simplest things to get right, and one of the easiest to break without noticing.
\nAn AI assistant answering \"how much is the Pro plan at northwind.example?\" or an agent asked to \"buy two bags of the house blend\" needs to get from your home page to the right page quickly. Agents differ in how they do that:
\nEither way, the home page is the map. If the pages that make you money aren't on it, the agent has to guess, search, or fall back on your sitemap, if it looks for one at all.
\nAgentScore goes looking for your key pages the way an agent would, in this order:
\n/pricing or /cart, for anything still missing.It recognizes a key page by its address or its link text. Addresses like /products/house-blend, /pricing, /plans and /cart count, and so do links labelled \"Pricing\", \"Plans\", \"Cart\", \"Basket\" or \"Bag\".
The check, \"Key pages are linked from the home page\", then asks one question: did the home page link to them directly?
\n| Type of site | Passes when the home page HTML links to |
|---|---|
| Online store | A product page or the cart |
| Software or subscription business | The pricing page |
| Neither found anywhere | Not tested |
If AgentScore only found your key pages through the sitemap or a common address, the check gives a warning: the pages exist, but an agent reading your home page wouldn't have been led to them. The same pages are then used for other checks, such as whether AI agents can open them and whether prices are readable, so linking them clearly helps more than one result.
\nMost sites that get a warning here do have links to their products and pricing. People can see them. The links just aren't where agents look.
\n<button> or <div> with a click handler goes nowhere for an agent that reads links. So does href=\"#\" with a script./landing/q4-campaign, gives neither people nor agents much to go on.Think of the questions people ask AI assistants about businesses like yours, and make sure the answer is one link away from the home page.
\n| Type of site | Link to from the home page |
|---|---|
| Online store | Main categories, a few bestselling products, the cart, shipping, returns, contact |
| Software company | Pricing, sign-up, product overview, docs, contact sales |
| Service business | Services and prices, booking, locations and hours, contact |
Footer links count. A plain footer with \"Pricing\", \"Shipping\" and \"Returns\" is often the most reliable route an agent has, because it's on every page and rarely depends on JavaScript.
\nThe fix is ordinary links, present in the HTML your server sends, with clear text:
\n<!-- Hard for agents: no link in the HTML, vague text -->\n<div class=\"nav-item\" onclick=\"openMenu('shop')\">Discover</div>\n<span class=\"cart-icon\" onclick=\"openCart()\"></span>\n<!-- Easy for agents: real links, plain names -->\n<nav aria-label=\"Main\">\n <a href=\"/collections/coffee\">Coffee</a>\n <a href=\"/collections/equipment\">Brewing equipment</a>\n <a href=\"/pricing\">Pricing</a>\n <a href=\"/cart\" aria-label=\"Cart\"><svg aria-hidden=\"true\">…</svg></a>\n</nav><a href> with a real address for anything that goes to another page. A cart drawer is fine, as long as the cart icon is also a link to a cart page./pricing, /cart and /products/house-blend tell an agent what a page is before it opens it.href=\"/cart\", /pricing or /products/. If they're not there, agents that read HTML can't see them.Your sitemap is a safety net, not a substitute. A good XML sitemap helps crawlers find every page, and llms.txt can point AI straight to your most important ones. But plenty of agents never read either. Links on the home page are the route every agent can follow.
\nOnce agents can reach your key pages, the next questions are whether they can read them and act on them. See prices AI agents can read, buttons, links and forms agents can use and site search and filters AI agents can use.
", "date_published": "2026-10-09T12:04:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Navigability" ] }, { "id": "https://ghostagentlab.com/articles/site-search-filters-ai-agents/", "url": "https://ghostagentlab.com/articles/site-search-filters-ai-agents/", "title": "Site search and filters AI agents can use", "summary": "Build a search box, results URLs and filters that AI agents can operate, so they find the right products instead of giving up or guessing.", "content_html": "Ask an AI agent to \"find a waterproof trail shoe under $150 in size 10\" and, on most stores, it does what a person would: types into the search box, then narrows the results with filters. If the search box is hard to find, the results only exist inside a script, or the filters are custom widgets it can't operate, the agent gives up or picks the wrong thing. This guide covers how to build search and filters that AI agents can use, and that work better for people too.
\nThere are two broad ways an AI agent gets to a list of products:
\n/search?q=trail+shoes or a filtered category page. AI assistants that fetch pages rather than click through them can only work this way: they can't type into a box, but they can open a link.A site that supports both gives every kind of agent a route to the right products. A site that supports neither forces agents to browse category pages one by one, which is slow and often ends with a worse answer.
\nThe search box should be a real form that works without anything clever. That means a labelled field, a named submit button, and a results page that loads when the form is submitted.
\n<form role=\"search\" action=\"/search\" method=\"get\">\n <label for=\"q\">Search products</label>\n <input id=\"q\" name=\"q\" type=\"search\" autocomplete=\"off\">\n <button type=\"submit\">Search</button>\n</form>aria-label such as \"Search products\". A magnifying-glass icon on its own has no name an agent can read.This is the single most useful change for AI agents, and it's good for people too: results can be bookmarked, shared and reached with the back button.
\nEvery search and every combination of filters should be reflected in the address, using readable parameters:
\nhttps://northwind.example/search?q=trail+shoes\nhttps://northwind.example/collections/shoes?waterproof=yes&size=10&max_price=150\nhttps://northwind.example/collections/shoes?sort=price-asc&page=2size=10 and color=black are guessable. f=a8x2k isn't.Filters are where many stores lose agents. They're often built as custom widgets that look good but aren't recognizable as checkboxes, options or buttons. The fix is to use the controls the browser already provides.
\n| Filter | Hard for agents | Easy for agents |
|---|---|---|
| Size, brand, feature | Clickable boxes made from <div>s | Checkboxes with labels, grouped in a <fieldset> with a <legend> |
| Color | Swatches with no text, only a background color | Checkboxes or buttons named \"Black\", \"Navy\", with the swatch as decoration |
| Price | A drag-only slider | \"Min\" and \"Max\" number fields, with or without a slider alongside |
| Sort order | A custom dropdown that opens on hover | A native <select>, or a list of real links |
| More results | Infinite scroll only | Numbered page links, or a \"Load more\" button backed by a ?page=2 link |
<fieldset>\n <legend>Size</legend>\n <label><input type=\"checkbox\" name=\"size\" value=\"9\"> 9 (14)</label>\n <label><input type=\"checkbox\" name=\"size\" value=\"10\"> 10 (22)</label>\n</fieldset>\n<label for=\"sort\">Sort by</label>\n<select id=\"sort\" name=\"sort\">\n <option value=\"relevance\">Most relevant</option>\n <option value=\"price-asc\">Price: low to high</option>\n</select>A few more details that save agents from guessing:
\naria-live region lets assistive tech, and agents reading the page structure, notice the change.Once your search has a predictable URL, you can describe it so agents don't have to discover it:
\nhttps://northwind.example/search?q={query}\" in your llms.txt file hands the pattern to any AI that reads it.WebSite describe its search with a SearchAction. Google retired the search box it used to show for this in 2024, so don't expect a search result feature from it, but it remains a standard, machine-readable description of your search URL.Watch your robots.txt. Many sites stop search engines indexing internal search results, which is sensible. But a blanket Disallow: /search for every bot also tells AI assistants that respect robots.txt not to open those pages when a person asks them to. Decide deliberately which agents that rule should apply to. See robots.txt for AI agents.
AgentScore doesn't have a separate check for search, but several of its checks cover the building blocks. On the home page, where most search boxes live, it checks that form fields are labelled, that buttons and links have names, and that clickable things are real buttons and links. \"Content loads without JavaScript\" catches a home page whose content only appears once scripts run, a sign that results and filters may be built the same way. The free scan doesn't run a shopping journey itself; a Ghost Agent test does, and a search box or filter that doesn't work shows up in its step-by-step replay.
\nSearch is also one of the journeys Ghost Agents can test for you. You choose the journey, such as \"search for a waterproof trail shoe, filter to size 10 and open the first result\", and a Ghost Agent works through it like a customer would, on the schedule you set. When a theme update or a new search app breaks it, you find out before customers do. See how to test your key journeys with AI agents.
\nFor the controls on the rest of the page, see buttons, links and forms agents can use. For getting agents to your categories in the first place, see link your key pages from the home page.
", "date_published": "2026-10-09T12:03:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Navigability" ] }, { "id": "https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/", "url": "https://ghostagentlab.com/articles/agent-friendly-sign-up-booking/", "title": "Sign-up, booking and lead forms AI agents can finish", "summary": "How to build sign-up, booking and quote forms AI agents can complete, and how to handle email verification, CAPTCHAs, date pickers and the handoff.", "content_html": "\"Book me a table for four on Friday.\" \"Get me a quote for a new roof.\" \"Sign me up for the free trial.\" More of these requests now go to an AI assistant instead of a search box, and the assistant has to finish the form on your site. If it can't, the lead goes to a competitor whose form it can. This guide covers sign-up, booking and lead forms, and the steps where agents most often get stuck.
\nReading a page is easy for an AI agent. Finishing a form is not. Every field has to be matched to something the person asked for, every choice has to be made, and every step has to confirm that it worked. Forms also tend to carry the things built to stop bots: CAPTCHAs, email verification and one-time codes.
\nNone of this means removing your defenses. It means designing the form so an agent can do the routine part, and handing over to the person at the moments that genuinely need them.
\ntype=\"email\", type=\"tel\", autocomplete=\"given-name\", postal-code and organization tell an agent exactly what each field wants.<select> menus, radio buttons and checkboxes, not custom dropdowns built from <div>s.For software companies, sign-up is the conversion. An agent asked to \"start a trial\" needs to find the sign-up page, fill it in and reach a confirmed account, or at least a clear next step for the person.
\nAn agent working on someone's behalf usually can't read their inbox or their text messages, and you wouldn't want it to have that access by default. So any step that says \"enter the code we just sent\" is a handoff point. Handle it well:
\nautocomplete=\"one-time-code\" and inputmode=\"numeric\" on a single field are easier for people and agents than six separate boxes.A CAPTCHA on every submission stops AI agents as surely as it stops spam bots. Most sites don't need one on every form. Better options:
\nAgentScore checks for a CAPTCHA or challenge when an agent arrives on your site, and at the cart and checkout. See bot protection and CAPTCHAs for the full picture.
\nBooking is where custom widgets cause the most trouble. A calendar that only works by clicking tiny cells, or a time-slot grid of unlabelled boxes, leaves an agent guessing.
\n| Element | Hard for agents | Easy for agents |
|---|---|---|
| Date | Custom calendar pop-up with no labels on days | <input type=\"date\">, or a calendar whose days are buttons named \"Friday 10 October, available\" |
| Time | Grid of colored boxes | Radio buttons or a list of buttons named \"7:30 pm\" |
| Availability | Grayed-out slots with no text | Disabled options with a reason: \"7:00 pm, fully booked\" |
| Party size or service | Plus and minus icons only | A <select> or number field with a label |
Also make availability readable as text where you can, for example a list of open times on the page. An assistant answering \"is there anything Friday evening?\" can then answer without driving the widget at all.
\n<fieldset>\n <legend>Time on Friday 10 October</legend>\n <label><input type=\"radio\" name=\"time\" value=\"19:00\" disabled> 7:00 pm (fully booked)</label>\n <label><input type=\"radio\" name=\"time\" value=\"19:30\"> 7:30 pm</label>\n <label><input type=\"radio\" name=\"time\" value=\"20:00\"> 8:00 pm</label>\n</fieldset>If your booking runs inside a third-party widget in an iframe, test it. Some booking providers build accessible widgets and some don't, and you inherit whatever they ship.
\nSome steps belong to the person: paying a deposit, accepting terms that bind them, confirming an email address, or signing in with a password only they know. Agents generally stop and hand back at these points, and that's how it should be. Your job is to make the handoff smooth:
\nA quick test: ask an AI assistant with browsing to complete your main form using test details, and watch where it hesitates. Then try the same journey with only the keyboard. The two usually get stuck in the same places. For a structured approach, see how to test your key journeys with AI agents.
\nAgentScore checks the parts of this you can see from the outside: form fields are labelled, buttons and links have names, pop-ups can be dismissed, there's no CAPTCHA on arrival, and agents can find your sign-up or add-to-cart button. Ghost Agents go further: they run the journey itself, such as sign-up or a contact form, with test data you supply, on a schedule, and show a step-by-step replay when it breaks. They don't submit payments, and the sign-up template stops short of email confirmation.
\nForms are only half of task completion. For stores, the other half is checkout and guest checkout.
", "date_published": "2026-10-09T12:02:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Task completion" ] }, { "id": "https://ghostagentlab.com/articles/test-journeys-with-ai-agents/", "url": "https://ghostagentlab.com/articles/test-journeys-with-ai-agents/", "title": "How to test your key journeys with AI agents", "summary": "Which journeys to test with AI agents, how to test by hand and with Ghost Agents, what to measure, and how to test safely without real purchases.", "content_html": "Technical checks tell you whether AI agents should be able to use your site. The only way to know whether they can is to send one through and watch. This guide covers which journeys to test, how to test them by hand with the assistants your customers already use, how to automate it, and how to keep testing safe.
\nA site can pass every check for labels, named buttons and structured data and still lose agents at a custom size picker, a cookie banner that reappears on the cart page, or a checkout step that needs an account. Those failures only show up when an agent actually tries to finish the job. That's why task completion carries the most weight in agent readiness, and why it can only be measured by running real agents.
\nDon't test everything. Start with the two or three journeys where an agent failure costs you a customer:
\n| Type of site | Journeys to test first |
|---|---|
| Online store | Search for a product, choose options and add to cart, check out as a guest up to payment |
| Software company | Find pricing, start a trial or sign up, book a demo |
| Service business | Find prices and availability, book an appointment, request a quote |
| Any site | Find a policy (returns, shipping, cancellation), use the contact form |
Write each one the way a customer would ask an assistant: \"Find a waterproof jacket in medium under $150 and add it to the cart.\" Specific goals give clearer results than \"buy something\".
\nThe quickest start costs nothing. Several consumer AI assistants can now browse and act on websites, usually in an \"agent\" or browsing mode. Use the ones your customers are likely to use.
\nManual testing is good for discovery: it shows you how agents experience your site and builds a shared picture for the team. It doesn't scale, and it won't tell you when something breaks next Tuesday.
\nGhost Agents are AI agents that run your key journeys for you. In the Ghost Agent Labs app, you set up a test for a site you've added:
\nEach run records every step: the page the agent saw, what it did and its reasoning. When a run fails, the replay opens on the step where it got stuck, and you can compare it with the last run that passed. Failing tests can alert your team by email or Slack.
\nFlaky is a finding, not noise. If an agent completes a journey two times in three, a real customer's assistant fails one time in three. Treat a flaky result as a problem to fix, and look at the failed runs to see what was different.
\nTesting with agents means letting software click buttons on your live site. Set some rules:
\nGhostAgent/1.0 in the user agent and sign their requests with Web Bot Auth, so they can be recognized and filtered from reports.A failed run is only useful if someone acts on it. For each failure, record the journey, the step, what the agent was trying to do and a screenshot, then route it to whoever owns that part of the site: the developer for an unlabelled button, the platform admin for forced account creation, marketing for the pop-up. The replay usually tells you which.
\nMost fixes are the same ones covered elsewhere in these guides: buttons and forms agents can use, cookie banners and pop-ups, checkout and sign-up and booking forms.
\nWhen someone asks ChatGPT, Perplexity or Gemini for a recommendation and clicks a link in the answer, they land on your site as a visitor like any other. Those visits can be measured in the analytics you already have, and they're often worth separating out. This guide shows how to spot them, how to report on them, and what the numbers can't tell you.
\nAI shows up in your data in two ways, and it helps to keep them apart:
\nThe first tells you whether AI can read and use your site. The second tells you whether that's turning into visits and sales.
\nAnalytics tools work out where a visit came from in two main ways: the referrer, which the browser sends to say which site the person came from, and campaign tags (UTM parameters) on the link itself.
\nWhen someone clicks a link in a web-based assistant, the browser often sends that assistant's domain as the referrer. Domains you're likely to see include:
\n| Assistant | Referrer domains you may see |
|---|---|
| ChatGPT | chatgpt.com (older visits may show chat.openai.com) |
| Perplexity | perplexity.ai, www.perplexity.ai |
| Google Gemini | gemini.google.com |
| Microsoft Copilot | copilot.microsoft.com |
| Claude | claude.ai |
Treat this list as a starting point rather than a complete map. Assistants change domains, launch new apps and change how they link out, so check your own referral report for new sources every month or so.
\nSome assistants add their own tag to outbound links. ChatGPT, for example, has been adding utm_source=chatgpt.com to many links in its answers. In your analytics that shows as the source chatgpt.com, sometimes with no medium, or alongside the referrer. Other assistants may do the same, or nothing at all.
Not every AI referral announces itself. Mobile and desktop apps often open links without sending a referrer, privacy settings and browsers can strip it, and a person who copies a URL from an answer and pastes it into a new tab arrives as \"direct\". Some of your direct traffic is AI referral traffic you can't see. Treat any AI referral number as a floor, not a total.
\nAnalytics tools have been changing how they group this traffic. In GA4's default channel grouping, AI assistants have typically ended up under Referral, mixed in with every other website that links to you. Check what your property shows today, because default channel definitions get updated. If AI referrals aren't already separated, two approaches work:
\nA source pattern to start from:
\n(^|\\.)(chatgpt\\.com|chat\\.openai\\.com|perplexity\\.ai|gemini\\.google\\.com|copilot\\.microsoft\\.com|claude\\.ai)$Use a \"matches regex\" condition on session source. Keep the pattern in one place, such as a shared doc, and add new assistants as they appear in your referral report. Other analytics tools work the same way: a rule on referrer or source that puts these visits in their own channel.
\nGoogle's own AI features are harder. Clicks from AI Overviews and AI Mode in Google Search generally arrive as ordinary Google organic traffic, and analytics can't separate them from classic search results. Search Console reports them as part of overall Search performance. Don't expect your \"AI assistants\" channel to include them.
\nBe honest with stakeholders about what this data can and can't show:
\nAssistants recommend and link to sites they can read and trust. The groundwork is the same as for agent readiness: let AI search and assistant agents in through robots.txt, put key content in the HTML, describe products and prices with structured data, and make policy pages easy to quote. If you block OAI-SearchBot or PerplexityBot, don't expect referrals from ChatGPT search or Perplexity.
Then make sure the visits convert. The person arriving from an AI answer, or the agent acting for them, needs a page that works. AgentScore shows how your site looks to AI agents, and Ghost Agents test the journeys that turn a referral into a sale.
", "date_published": "2026-10-09T12:00:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Measurement" ] }, { "id": "https://ghostagentlab.com/blog/next-customer-not-human/", "url": "https://ghostagentlab.com/blog/next-customer-not-human/", "title": "Your next customer won't be human", "summary": "AI agents now read, compare and buy on people's behalf. Most websites can't see them, don't know if they succeed, and were never built for them. Here's what's changing.", "content_html": "For twenty-five years, websites have been built for two kinds of visitor: people, and the search crawlers that send people. A third kind is arriving fast. AI agents now read, compare, book and buy on people's behalf, and most websites can't see them, don't know whether they succeed, and were never designed for them.
\nWhen someone asks an AI assistant \"which of these running shoes is best for flat feet, and is it in stock in a 10?\", a person never visits your site. Software does. That software comes in three broad kinds:
\nThe last two are the ones that matter most to revenue, because each visit stands in for a real customer with a real intent. If the agent can't find the price, read the return policy, or get past a pop-up, the customer doesn't get a worse experience on your site. They get a recommendation for someone else's.
\nMost websites have no idea how many agents visit, for a simple reason: analytics tools count visitors with a JavaScript tag, and most agents never run it. Crawlers and assistant fetchers request the HTML and leave. Your dashboards show a quiet day while your server logs show a busy one.
\nThe agents that do run JavaScript often look like ordinary browsers, so they blend into the human numbers. Either way, the question \"did the agent get what it came for?\" goes unanswered.
\nWhen we test sites with real agents, the failures fall into the same few patterns:
\nNone of these show up in a normal QA pass, because a person gets through every one of them without noticing.
\nTwo kinds of tools touch this problem, and neither solves it.
\nBot management decides who to block. It's essential, and it's built to keep bad traffic out, which means good agents are often collateral damage. AI visibility tools track what chatbots say about your brand. That's useful too, but it measures the answer, not whether an agent could use your site to get there.
\nWhat's missing is the thing we've always done for human visitors: test the experience. Can an agent actually complete the journeys that make you money, and does it still work after this week's release?
\nWe think agent experience will become a discipline of its own, next to user experience and SEO. It has three parts, and they're how we've built Ghost Agent Labs:
\nIt's the same idea behind Ghost Inspector, which has tested websites for human visitors for years, applied to the visitors that aren't human.
\nYour next million visitors won't be human. The sites that are ready for them will win the customers they bring.
", "date_published": "2026-10-08T12:11:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Perspective" ] }, { "id": "https://ghostagentlab.com/blog/signed-requests/", "url": "https://ghostagentlab.com/blog/signed-requests/", "title": "Why we sign every request our agents send", "summary": "Anyone can copy a user agent. Here's how Ghost Agent Labs signs its scanner and Ghost Agent traffic with Web Bot Auth, and why every well-behaved agent should.", "content_html": "A user agent string is a name tag anyone can print. If AI agents are going to shop, book and sign up on people's behalf, websites need a better way to tell who's knocking. That's why every request our scanner and Ghost Agents send under their own name is cryptographically signed.
\nEvery request a browser or bot makes carries a User-Agent header. Ours look like this:
AgentScore/1.0 (+https://ghostagentlab.com/agentscore/bot)\nGhostAgent/1.0 (+https://ghostagentlab.com/ghost-agent)That's useful, and it's a courtesy we'll keep. But it proves nothing. A scraper can send GPTBot, ClaudeBot or AgentScore/1.0 just as easily as the real thing, and plenty do. So site owners face a bad choice: trust the label and let impostors in, or block the label and turn away the agents they actually want.
The traditional fix is to check where the request came from: published IP ranges, or a reverse DNS lookup that resolves back to the operator's domain. That works for big crawlers with fixed infrastructure. It works badly for browser agents running in the cloud, where IP addresses change all the time, and it puts the burden on every website to keep lists up to date.
\nWeb Bot Auth is a proposal, now being worked on at the IETF and supported by Cloudflare's verified bots program, that lets an agent prove who it is the same way websites prove who they are: with a public key.
\nhttps://app.ghostagentlab.com/.well-known/http-message-signatures-directory.Signature-Agent (where to find the keys), Signature-Input (what was signed, when, and with which key) and Signature (the signature itself), following HTTP Message Signatures, RFC 9421.Anyone can copy a user agent. Only the holder of the private key can produce the signature.
\nWe sign with an Ed25519 key. Each signature covers the site's host name and our Signature-Agent header, carries a fresh random nonce, and expires after 60 seconds, so a captured signature can't be replayed against another site or reused later. The key ID is the key's standard JWK thumbprint, and the key directory itself is signed too, so nobody can swap in their own keys.
| Requests | Signed? |
|---|---|
| AgentScore requests under its own user agent: robots.txt, sitemaps, well-known files, page discovery | Yes |
| Every Ghost Agent request, including images and scripts from other hosts | Yes |
| AgentScore requests that deliberately imitate a browser or another agent, such as ChatGPT-User, to see how your site treats them | No, so your site treats them exactly as it would that visitor |
That last row matters. Part of what AgentScore measures is whether your bot protection treats AI agents differently from people. If we signed those requests, a CDN that trusts us would wave them through and the test would tell you nothing.
\nAgents are becoming customers' representatives. When someone asks an assistant to reorder printer ink or book a table, the agent that arrives at your site is acting for a real person with a real wallet. The sites that win are the ones that can say yes to those agents confidently, without opening the door to every scraper wearing a borrowed name.
\nSigned requests make that possible. A site can allow verified agents through a challenge page, give them a lighter rate limit, or simply count them accurately in analytics. None of that works if the identity is a string anyone can type.
\nIf you run an agent: sign your requests. Publish a key directory, add the three headers, and register with the CDNs your users' sites sit behind. It's a small amount of work, and it's the difference between being trusted and being blocked.
\nIf you run a website: check whether your CDN or bot protection can verify Web Bot Auth signatures, and prefer it over IP allowlists for the agents you want. Then run AgentScore to see how your site treats AI agents today.
\nThe details of what our agents do, what they won't do, and how to opt out are on the AgentScore bot and Ghost Agent pages. We don't use fixed IP addresses yet, so if you want to allow us, please do it by signature or user agent rather than by IP.
", "date_published": "2026-10-08T12:10:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Blog", "Engineering" ] }, { "id": "https://ghostagentlab.com/articles/agent-readiness/", "url": "https://ghostagentlab.com/articles/agent-readiness/", "title": "What is agent readiness? A complete guide", "summary": "How well AI agents can get into your site, understand it, find their way around and complete tasks: what each part means, what good looks like, and where to start.", "content_html": "Agent readiness is how well AI agents can get into your website, understand it, find their way around it, and complete the tasks people send them to do. This guide explains each part, what good looks like, and where to start.
\nAI assistants increasingly visit websites on a person's behalf: to answer a question, compare products, check a policy, or buy something. Each of those visits stands in for a customer. If the agent can't use your site, it doesn't complain or call support. It moves on to a site it can use, and recommends that one instead.
\nAgent readiness is not the same as SEO, though they overlap. SEO is about being found and ranked. Agent readiness is about what happens next: whether an agent that has found you can actually do something useful with your site.
\nNot every agent wants the same thing. A crawler collecting pages for AI training is very different from an assistant checking your returns policy for a customer, or a browser agent trying to check out. See the AI agents that visit your website, and keep the agent readiness glossary handy for the terms in this guide.
\nAgentScore measures agent readiness out of 100, across four categories. The weights reflect what stops agents most often in practice.
\n| Category | The question | Points |
|---|---|---|
| Access | Can agents get in? | 25 |
| Readability | Can agents understand your pages? | 25 |
| Navigability | Can agents find their way around? | 20 |
| Task completion | Can an agent actually get the job done? | 30 |
Before anything else, the agent has to be allowed through the door. Most access failures are accidental: a robots.txt rule copied from a template, or bot protection that challenges every visitor that isn't a person.
\nMany agents read the HTML your server sends and never run JavaScript. If your prices, product details or policies only appear after scripts load, those agents see an empty page.
\nBrowser agents find their way around a page much as a screen reader does: by the names and roles of buttons, links and form fields. Much of agent readiness here is simply good accessibility. See buttons, links and forms agents can use.
\nThis is the part that matters most, and the only way to measure it is to try. A site can pass every technical check and still lose agents at the size selector or the checkout button.
\nThe free AgentScore scan doesn't test this yet: it marks Task completion “not tested” and works out your score from the other three categories. Ghost Agents test it today, running the journeys you choose on a schedule, with a step-by-step replay of each run, and alerting you when one breaks.
\n/robots.txt and make sure it doesn't block AI assistants or AI search agents.No. Agent readiness is about letting in the agents you want, and the main ones can be verified. Keep blocking unknown and abusive bots; just make sure your rules don't catch AI assistants that are bringing you customers.
\nNo. Training crawlers and the agents that fetch pages for people use different names, so you can block one and allow the other. Our robots.txt guide shows how.
\nThere's a big overlap, especially in navigation, and work on one helps the other. But agent readiness also covers access rules, machine-readable data, and whether an agent can finish a real task.
", "date_published": "2026-10-08T12:09:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Start here" ] }, { "id": "https://ghostagentlab.com/articles/robots-txt-ai-agents/", "url": "https://ghostagentlab.com/articles/robots-txt-ai-agents/", "title": "robots.txt for AI agents: who to allow and who to block", "summary": "The three kinds of AI agent, the robots.txt names each one uses, a sensible default, and the most-specific-group rule that catches most people out.", "content_html": "Your robots.txt file may be blocking the AI assistants that send you customers, while doing nothing to stop the bots you actually wanted to keep out. This guide explains the three kinds of AI agent, which robots.txt names each one uses, and how to write rules that let the right ones in.
\n\"AI bots\" isn't one thing. The major AI companies each run several agents with different jobs and different names, and the right policy for each is different.
\n| Kind | What it does | Examples (robots.txt name) |
|---|---|---|
| Training crawlers | Collect pages to train AI models | GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent |
| AI search crawlers | Index pages so AI search can cite and link to them | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot |
| Assistant fetchers | Fetch a page in real time because a person asked about it | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, DuckAssistBot, MistralAI-User |
Some names aren't crawlers at all. Google-Extended and Applebot-Extended are control tokens: Google and Apple crawl with Googlebot and Applebot as usual, and these names let you say whether that content may be used for their AI models. Blocking them doesn't affect normal search.
Lists like this change as companies launch new agents, so check each operator's documentation for the current names.
\nFor most businesses, the agents that matter most are AI search and assistant fetchers, because they answer people's questions about you and link back to you. Training is a separate decision about your content, and it's yours to make.
\nThis robots.txt allows everything, which is also what you get with no robots.txt at all:
\nUser-agent: *\nAllow: /\nSitemap: https://www.example.com/sitemap.xmlIf you'd rather not have your content used to train AI models, but still want to appear in AI search and assistant answers, block only the training names:
\n# Allow AI search and assistants; opt out of AI training\nUser-agent: GPTBot\nUser-agent: ClaudeBot\nUser-agent: CCBot\nUser-agent: Google-Extended\nUser-agent: Applebot-Extended\nUser-agent: meta-externalagent\nDisallow: /\nUser-agent: *\nDisallow: /cart\nDisallow: /account\nAllow: /\nSitemap: https://www.example.com/sitemap.xmlListing several User-agent lines above one set of rules applies those rules to all of them.
A crawler follows only the most specific group that matches its name, and ignores the rest. If there's a group for GPTBot, GPTBot ignores everything under User-agent: *.
So in the example above, the training crawlers don't inherit the /cart and /account rules; they don't need them, because they're blocked from everything. But the reverse mistake is common:
User-agent: *\nDisallow: /checkout\nDisallow: /admin\nUser-agent: OAI-SearchBot\nAllow: /Someone added that second group to \"make sure\" OpenAI's search crawler gets in. Instead, they've told it that /checkout and /admin are open, because it no longer reads the * group. If you add a group for a specific agent, copy into it every rule you want it to follow.
Disallow: /. Staging sites are often launched with a robots.txt that blocks everything. Check yours today./products or /pricing for all bots, or for *, to save crawl budget also hides them from agents.When a person asks an assistant to read a specific page, some operators treat that fetch like a person clicking a link rather than a crawl, and say robots.txt rules may not apply to it. If you need to stop those visits completely, that has to happen at your CDN or server, not in robots.txt. For most businesses, though, these are the visits you want most: each one is a customer asking about you.
\nhttps://yourdomain.com/robots.txt and read it top to bottom. Note every group and which names it applies to.robots.txt is only the first door. Next, make sure your bot protection lets the real agents through, and can tell them apart from impostors using their names. See how to tell if an AI crawler is real.
", "date_published": "2026-10-08T12:08:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Access" ] }, { "id": "https://ghostagentlab.com/articles/verify-ai-crawlers/", "url": "https://ghostagentlab.com/articles/verify-ai-crawlers/", "title": "How to tell if an AI crawler is real", "summary": "Anyone can claim to be GPTBot or Googlebot. How to verify AI agents with published IP ranges, reverse DNS and signed requests, and what to do when there's no proof.", "content_html": "Anyone can send a request that says it's GPTBot or Googlebot. To let real AI agents in while keeping impostors out, you need to check where a request actually came from. There are three ways to do it, and the right one depends on the operator.
\nEvery request names its sender in the User-Agent header, and that name is a promise, not a proof. Scrapers routinely borrow the names of well-known crawlers, because many sites let those crawlers through without a challenge. If your bot rules trust the name alone, you're letting in anyone who copies it. If they distrust it, you're blocking real agents along with the fakes.
Verification resolves that. It answers one question: did this request really come from the company it names?
\nSeveral operators publish the IP addresses their crawlers use, as machine-readable JSON files. If a request claims to be one of their agents and comes from an address in the list, it's real.
\n| Operator | Agents covered |
|---|---|
| Googlebot, special-case crawlers and user-triggered fetchers (separate lists) | |
| Microsoft | Bingbot |
| Apple | Applebot |
| OpenAI | GPTBot, OAI-SearchBot and ChatGPT-User (one list each) |
| Perplexity | PerplexityBot and Perplexity-User (one list each) |
Each operator links its list from its crawler documentation. A few practical points:
\nSome operators instead promise that their crawlers' IP addresses resolve to a host name on their own domain. The check has two steps, because reverse DNS on its own can be faked by whoever controls the IP address:
\nA request arrives
\nUser agent says Googlebot, from IP address 66.249.66.1
1. Reverse lookup
\n66.249.66.1 → crawl-66-249-66-1.googlebot.com
Ends in googlebot.com
\n2. Forward lookup
\ncrawl-66-249-66-1.googlebot.com → 66.249.66.1
Points back to the same address
\nVerified
\nThe request is from Google. If either step fails, it isn’t, whatever its user agent says.
\nGoogle's own example, from a terminal:
\n$ host 66.249.66.1\n1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.\n$ host crawl-66-249-66-1.googlebot.com\ncrawl-66-249-66-1.googlebot.com has address 66.249.66.1The name ends in googlebot.com and points back to the same address, so the request is from Google. Domains that operators document for this include:
| Agent | Host name ends in |
|---|---|
| Googlebot and other Google crawlers | googlebot.com, google.com or googleusercontent.com |
| Bingbot | search.msn.com |
| Applebot | applebot.apple.com |
| Amazonbot | crawl.amazonbot.amazon |
DNS lookups are slow compared with a page request, so cache the result for each IP address rather than checking on every hit.
\nThe newest method doesn't depend on IP addresses at all. With Web Bot Auth, the agent signs each request with a private key and publishes the matching public key on its own domain. Your CDN or server checks the signature, and a valid one proves the sender, whatever network it came from.
\nThat makes it the best fit for browser agents running in the cloud, whose addresses change constantly. It's still early, so check whether your CDN supports it. We explain how it works, and why we sign all of our own agents' requests, in Why we sign every request our agents send.
\nNot every operator publishes IP ranges, a reverse DNS domain, or signing keys for every agent. For those, you can't prove a request is real or fake; you can only weigh the evidence, such as how it behaves and how fast it requests pages.
\nTreat \"can't tell\" as its own answer, not as \"fake\". Blocking everything you can't verify will block legitimate agents from operators that simply haven't published proof yet.
\nYour CDN may already do this. Many CDNs and bot-management products maintain lists of verified bots. Check that the AI assistants and AI search agents you want are in the allowed categories, and that \"AI crawler\" blocking rules aren't catching them by name.
\nGhost Agent Labs does it for you. It identifies every agent in your server or edge traffic, checks it against published IP ranges and reverse DNS, and flags impostors, so you can see which agents are real before you decide what to allow. Missing or stale data is reported as \"can't tell\", never as \"spoofed\". Start free.
\nBot protection exists to keep out scrapers, credential stuffers and fraud. But the same rules often stop the AI assistants trying to answer a customer's question about your products. This guide shows how to tell whether that's happening, and how to tune your defenses so the agents you want get through and the bots you don't still get stopped.
\nBot protection, whether it's a CDN feature, a web application firewall (WAF) or a dedicated product, looks for signs that a visitor isn't a person. AI agents show many of those signs, for entirely legitimate reasons:
\nThe quickest test is AgentScore. It requests your home page as a normal browser and as five AI agents, including ChatGPT-User, Claude-User, Perplexity-User and OAI-SearchBot, and compares what comes back. It fails the check if an agent is blocked or challenged while the browser gets through, and warns if an agent gets less than half the content the browser does. Separate checks look for CAPTCHAs on your home page, and test whether agents can open your product, pricing and cart pages.
\nThe safe way to let agents in is by verified identity. Most bot management products keep a list of verified bots, checked against the operator's published IP ranges, reverse DNS or signatures, and let you allow categories of them. Allow the categories for AI assistants and AI search, and keep blocking requests that only claim those names. Our guide to telling real AI crawlers from fakes explains how verification works.
\nNever allow by user agent alone. A rule like \"if the user agent contains GPTBot, skip all checks\" is the first thing scrapers exploit.
\nIf you've turned on a \"block AI bots\" setting, check exactly which agents it covers. If your goal is to opt out of AI training, block only training crawlers, and leave assistant fetchers and AI search crawlers allowed. See robots.txt for AI agents for which names are which.
\nAbuse mostly targets actions: logging in, creating accounts, applying discount codes, checking gift card balances, submitting payment. It rarely needs protecting against on pages that only display information. Concentrate strict rules and challenges on:
\nKeep product, category, pricing, policy and help pages as open as you safely can. Those are the pages agents need to answer questions about you.
\nA real assistant fetcher makes a handful of requests because a person asked about you. A scraper makes thousands. A sensible rate limit per IP address or verified agent stops the scraper without punishing the assistant, where a blanket block stops both.
\nAI agents can't solve CAPTCHAs, and aren't meant to. If you need a challenge, use invisible or risk-based challenges that only escalate when traffic looks abusive, and keep them off the pages agents need to read. Never put a CAPTCHA in front of ordinary content on arrival.
\nSome setups don't block agents outright, but quietly serve them a stripped-down page, a placeholder, or an \"enable JavaScript\" message. To the agent this is as bad as a block, and it's harder to notice. Agents should get the same content a browser does.
\nTo keep an eye on this over time, watch how often agents are blocked in your traffic. See how to measure AI agent traffic.
", "date_published": "2026-10-08T12:06:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Access" ] }, { "id": "https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/", "url": "https://ghostagentlab.com/articles/structured-data-ai-shopping-agents/", "title": "Product data that AI shopping agents can read", "summary": "How to describe products, prices, stock, shipping and returns with schema.org structured data, with a complete JSON-LD example and the rules that keep it trustworthy.", "content_html": "When an AI agent compares products for someone, it needs facts it can trust: the exact name, the price, the currency, whether it's in stock, how shipping and returns work. Structured data hands it those facts in a standard format, instead of making it guess from your page layout.
\nStructured data is a small block of machine-readable information in your page's HTML that describes what the page is about, using the shared vocabulary at schema.org. The most common format is JSON-LD: a <script type=\"application/ld+json\"> tag that people never see, but search engines and agents read directly.
Search engines have used it for years to show prices and star ratings in results. For AI agents it matters even more. A person can glance at a page and see that \"$24\" next to the \"Add to cart\" button is the price. An agent reading the raw page has to infer that, and on a busy page with sale prices, bundle prices and \"from\" prices, it can get it wrong. Structured data removes the guesswork.
\nHere's JSON-LD for a single product page, with the details agents use most:
\n<script type=\"application/ld+json\">\n{\n \"@context\": \"https://schema.org\",\n \"@type\": \"Product\",\n \"name\": \"Ethiopia Yirgacheffe Whole Bean Coffee, 340 g\",\n \"description\": \"Light roast single-origin coffee with notes of jasmine and lemon.\",\n \"sku\": \"NW-ETH-340\",\n \"gtin13\": \"0123456789012\",\n \"brand\": { \"@type\": \"Brand\", \"name\": \"Northwind Coffee\" },\n \"image\": \"https://northwind.example/images/ethiopia-340.jpg\",\n \"url\": \"https://northwind.example/products/ethiopia-yirgacheffe\",\n \"aggregateRating\": {\n \"@type\": \"AggregateRating\",\n \"ratingValue\": \"4.7\",\n \"reviewCount\": \"212\"\n },\n \"offers\": {\n \"@type\": \"Offer\",\n \"price\": \"18.00\",\n \"priceCurrency\": \"USD\",\n \"availability\": \"https://schema.org/InStock\",\n \"itemCondition\": \"https://schema.org/NewCondition\",\n \"url\": \"https://northwind.example/products/ethiopia-yirgacheffe\",\n \"hasMerchantReturnPolicy\": {\n \"@type\": \"MerchantReturnPolicy\",\n \"applicableCountry\": \"US\",\n \"returnPolicyCategory\": \"https://schema.org/MerchantReturnFiniteReturnWindow\",\n \"merchantReturnDays\": 30\n }\n }\n}\n</script>| Field | Why agents need it |
|---|---|
name | The exact product, including size or pack where it matters |
offers.price and priceCurrency | The price the customer will actually pay, as a plain number, with no currency symbol |
offers.availability | Whether it can be bought now. Agents skip products they think are out of stock |
sku, gtin, mpn | Let an agent match your product to the same item elsewhere when comparing |
brand | Answers \"do you have anything from …?\" questions |
aggregateRating | Lets agents weigh quality, but only include it if the reviews are on the page |
hasMerchantReturnPolicy, shippingDetails | Answer the questions people ask before buying, without a trip to your policy pages |
If a product comes in sizes or colors with different prices or stock, describe each variant. Schema.org's ProductGroup type groups variants under one parent, with variesBy naming what changes (such as size or color) and hasVariant listing each Product with its own offer. At minimum, make sure the price and availability on the page match the variant a visitor lands on.
Organization (or LocalBusiness with address and opening hours) with your name, logo, website, contact details and social profiles. AgentScore looks for structured data on your home page.BreadcrumbList shows where a page sits in your catalog, and helps agents move between categories.FAQPage for genuine question-and-answer content, such as shipping and returns questions.SoftwareApplication or Product with an Offer for each plan.application/ld+json.availability, identifiers like GTIN, and return and shipping details. Platform apps and SEO plugins can add these, or your developer can extend the theme template.Structured data is one part of being readable to agents. The other big one is making sure your content is in the HTML at all; see the guide to JavaScript-only content, or the full agent readiness guide.
", "date_published": "2026-10-08T12:05:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/javascript-content-ai-agents/", "url": "https://ghostagentlab.com/articles/javascript-content-ai-agents/", "title": "Why AI agents can't see your JavaScript-only content", "summary": "Many AI agents read only the HTML your server sends. How to check what they see, what usually goes missing, and how to fix it with server rendering.", "content_html": "Open a modern website with JavaScript turned off and you'll often see a logo, a spinner and not much else. That's what many AI agents see too. If your prices, products or policies only appear after scripts run, a large share of AI traffic never sees them.
\nWhen a browser loads a page, it does two things: it downloads the HTML your server sends, then runs the JavaScript, which may fetch more data and build much of what you see. Agents differ in how far they go:
\n| Agent | Typically reads |
|---|---|
| Training and AI search crawlers | The HTML only. Running JavaScript for billions of pages is slow and expensive, so most AI crawlers don't. |
| Assistant fetchers (fetching a page because someone asked) | Usually the HTML only, and they have a few seconds at most. |
| Browser agents | The full page, including JavaScript, but they're slow and impatient. Long loading states and content that appears late still trip them up. |
Googlebot does run JavaScript, which is why many sites that rely on it rank fine in Google search. That has hidden the problem: the same site can be well indexed by Google and nearly blank to AI assistants.
\ncurl -s https://yourdomain.com/your-page | grep -i \"your price\" shows exactly what your server sends.<div id=\"root\"> and everything else is built in the browser.The aim is simple: the important content should be in the HTML your server sends. JavaScript can still make the page interactive afterwards.
\nThe server builds the full HTML for each request, then JavaScript takes over in the browser. Modern frameworks support this directly: Next.js and Remix for React, Nuxt for Vue, SvelteKit for Svelte, and Angular's server rendering. If your site is a single-page app built with one of these, turning on server rendering for key pages is usually the biggest single fix.
\nPages are built to HTML ahead of time, at deploy. It's ideal for content that changes rarely, such as marketing pages, docs, policies and blog posts. Astro, Eleventy, Hugo and the frameworks above can all do it, often alongside server rendering for pages that change often.
\nYou don't need to render everything on the server. Prioritize what agents ask about: product name, price, availability, key specifications, shipping and returns, and plan pricing. Personalized recommendations, recently viewed items and chat widgets can stay in JavaScript.
\nPre-rendering services run your JavaScript ahead of time and serve the finished HTML to bots. It can work as a short-term fix, but it means maintaining two versions of each page, and it depends on correctly recognizing every agent. Google describes serving bots differently (\"dynamic rendering\") as a workaround rather than a long-term solution. Server rendering is better.
\n?page=2) agents can follow.<title> and <h1> in the HTML, never only after JavaScript runs.Once your content is in the HTML, the next question is whether agents can find their way around it and use it. See buttons, links and forms agents can use.
", "date_published": "2026-10-08T12:04:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Readability" ] }, { "id": "https://ghostagentlab.com/articles/agent-friendly-buttons-forms/", "url": "https://ghostagentlab.com/articles/agent-friendly-buttons-forms/", "title": "Buttons, links and forms agents can use: a checklist", "summary": "Browser agents navigate by the names and roles of controls. Fix icon-only buttons, fake buttons, unlabelled fields, blocking pop-ups and hover-only menus.", "content_html": "Browser agents don't see your page the way people do. They read its structure: what each button, link and form field is called, and what it does. A shopping-bag icon that a person recognizes instantly can be completely invisible to an agent. This checklist covers the fixes that matter most.
\nMost browser agents work from the page's accessibility tree, the same structured view of the page that screen readers use, often alongside a screenshot. In that tree, every element has a role (button, link, text box, heading) and a name (\"Add to cart\", \"Search\", \"Email address\"). The agent decides what to do by reading those names.
\nThat's good news: almost everything that makes a site work for agents also makes it work for people using screen readers and keyboards. If you've invested in accessibility, you're most of the way there.
\nWhat an agent reads
\nbutton with no name (the search icon)button with no name (the cart icon)heading “Trail runner”text “$129”generic: the color swatches aren’t controls at allbutton “Add to cart”The most common failure is the icon-only control: a cart, search, menu or close button that shows an icon and has no text.
\n<!-- An agent sees: button, no name -->\n<button><svg>…</svg></button>\n<!-- An agent sees: button \"Cart, 2 items\" -->\n<button aria-label=\"Cart, 2 items\"><svg aria-hidden=\"true\">…</svg></button>aria-label where the design calls for an icon alone.alt text becomes the link's name.A <div> with a click handler looks like a button to a person but isn't one to an agent: it has no role, can't be focused, and often isn't recognized as clickable at all.
<!-- Looks clickable, isn't recognized as clickable -->\n<div class=\"btn\" onclick=\"addToCart()\">Add to cart</div>\n<!-- Recognized as a button by every agent -->\n<button type=\"button\" onclick=\"addToCart()\">Add to cart</button><a href> for anything that goes to another page, and <button> for anything that does something on this page.role=\"button\" and tabindex=\"0\", and handle the Enter and Space keys.href=\"#\" and a script gives an agent nowhere to go if the script doesn't run.Agents fill in forms by matching what they've been asked to enter (\"my email\", \"ship to 10001\") to the field's label. Placeholder text is a weak substitute: it disappears once someone types, and not every agent reads it.
\n<label for=\"email\">Email address</label>\n<input id=\"email\" name=\"email\" type=\"email\" autocomplete=\"email\" required><label> to its field with for and id, or wrap the field in the label.type (email, tel, number) and autocomplete values (given-name, postal-code, address-line1). They tell an agent exactly what each field wants.<select>, checkboxes and radio buttons over custom-built dropdowns. Custom size and color pickers are a frequent place agents get stuck.aria-describedby, so an agent knows what to fix. A red border alone isn't enough.Cookie banners, newsletter pop-ups and region selectors that cover the page on arrival are one of the biggest blockers. An agent has to work out how to dismiss them before it can do anything else, and if it can't, it's stuck.
\nAgents click; they don't hover. A menu that only opens when the mouse rests over it may never open for an agent, hiding every category inside it.
\naria-expanded showing whether it's open.Landmarks tell an agent where the main content is, and where the navigation is, so it can skip straight to what matters.
\n<header>…</header>\n<nav aria-label=\"Main\">…</nav>\n<main>…</main>\n<footer>…</footer>Use one <h1> per page and headings in order, so the page has an outline an agent can follow.
Test the journeys, not just the page. A home page can pass every check while the size picker on a product page or the checkout button stops agents cold. Ghost Agents run your key journeys, such as search, add to cart and sign-up, on a schedule, and alert you when one breaks.
\nFor the bigger picture, see What is agent readiness?
", "date_published": "2026-10-08T12:03:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Navigability" ] }, { "id": "https://ghostagentlab.com/articles/agent-ready-checkout/", "url": "https://ghostagentlab.com/articles/agent-ready-checkout/", "title": "How to make checkout work for AI shopping agents", "summary": "A step-by-step guide to the shopping journey for AI agents: finding products, choosing options, adding to cart, checkout and the payment handoff.", "content_html": "When a person asks an AI assistant to buy something, the agent has to do everything a shopper does: find the product, choose the right options, add it to the cart, and get through checkout. Each step is a place it can get stuck, and a stuck agent is a lost sale. This guide walks the journey step by step.
\nAgents buy in two ways, and you'll want to support both:
\nProtocols are growing fast, but for now most agents still shop through the website. That's the journey this guide covers.
\nFind the product
\nHover-only menus, and search that only works with scripts
\nChoose options Most common failure
\nCustom size and color swatches with no names or roles
\nAdd to cart
\nA button covered by a banner, or no text confirming the add
\nThe cart
\nQuantity and remove buttons shown only as icons
\nCheckout
\nForced accounts, unlabelled fields and a CAPTCHA on every order
\nPayment Handed to the person
\nThe agent hands back to the person; a cart that empties during the handoff loses the sale
\n/search?q=decaf), lets agents search directly.<a href> links in the HTML, reachable without hover menus. See buttons, links and forms agents can use.This is where agents most often fail on otherwise good sites. Size, color and quantity pickers are frequently custom-built swatches that look fine to people but have no names or roles agents can read.
\n<select> elements or radio buttons, or give custom swatches a role and a name, for example a radio group with each option labelled \"Size 10\" or \"Color: Navy\".?variant=…), so an agent can link straight to the exact item.<button> with clear text, and nothing covers it, such as a cookie banner or chat widget./cart, that agents can open directly.autocomplete=\"shipping street-address\", postal-code, email and tel tell an agent exactly where each detail goes.Most agents stop at payment and hand back to the person, and that's how it should be. Make the handoff smooth: a clear payment step, wallet options like Apple Pay, Google Pay or Shop Pay that a person can approve in one tap, and a cart that survives the handoff instead of emptying after a short timeout.
\nTo let assistants complete purchases on their own, ask your commerce platform or payment provider which agentic commerce protocols it supports. AgentScore checks whether your site advertises one.
\nRetest after every change. Checkout breaks for agents in the same ways it breaks for people: a theme update, a new app, a redesigned size picker. The difference is that agents don't email support. They just buy elsewhere.
\nAsk most businesses how many AI agents visited their site last month and they can't say. It's not that the agents aren't coming. It's that the tools most teams use to count visitors were built in a way that can't see them. Here's why, and how to measure agent traffic properly.
\nTools like Google Analytics count a visit when a small JavaScript tag runs in the visitor's browser and reports back. That works for people. It fails for AI agents for three reasons:
\nYou may see AI in your analytics as referral traffic: people who click a link in a ChatGPT or Perplexity answer and arrive on your site. That's valuable, but it's the result of agent visits, not the visits themselves. It doesn't show which agents read your pages, which they couldn't, or what they saw.
\nEvery request to your site, from a person or an agent, passes through your web server or CDN, and they can log it whether or not any JavaScript runs. Each log line typically includes:
\n| Field | What it tells you |
|---|---|
| User agent | Which agent it claims to be |
| IP address | Where it came from, which you need to verify it's real |
| Path | Which page or file it asked for |
| Status code | Whether it got the page (200), was blocked (403), rate limited (429) or hit a missing page (404) |
| Time and bytes | When it came, and how much content it received |
If you have an Nginx or Apache access log in the standard \"combined\" format, this counts requests from some well-known AI agents:
\nawk -F'\"' '{print $6}' access.log \\\n | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|Amazonbot|meta-external[a-z]+|Bytespider|CCBot' \\\n | sort | uniq -c | sort -rnIt's a useful first look, but it has real limits: it trusts the user agent, so impostors count as the real thing; it only knows the names you list; and it says nothing about whether the agents succeeded.
\nLogs contain IP addresses, which are personal data for human visitors. You only need the full detail for bots and agents. Hash or drop IP addresses for human traffic, keep agent records separately, and set a retention period that matches your privacy policy.
\nGhost Agent Labs does this for you. Connect Cloudflare Logpush, a Vercel log drain, or any JSON log feed. Every request is matched against a registry of more than 1,500 known agents, crawlers are verified against published IP ranges and reverse DNS, and you get a dashboard of who's visiting, which agents are real, what they read, and where they're blocked. Human visits are only counted, never stored. Start free.
\nMeasuring traffic tells you who's coming. To find out whether they succeed, test the journeys that matter. See how to make checkout work for AI shopping agents.
", "date_published": "2026-10-08T12:01:00+00:00", "authors": [ { "name": "Ghost Agent Labs" } ], "tags": [ "Article", "Measurement" ] }, { "id": "https://ghostagentlab.com/articles/llms-txt/", "url": "https://ghostagentlab.com/articles/llms-txt/", "title": "How to write an llms.txt file", "summary": "What llms.txt is, what to put in it, a complete example for an online store, and the mistakes that make it useless to AI agents.", "content_html": "An llms.txt file is a short Markdown guide to your website, written for AI. It tells an agent what your business does and where the pages that matter are, so it doesn't have to work it out from menus, pop-ups and marketing copy. It takes about half an hour to write, and AgentScore checks for it.
\nllms.txt is a proposed standard, first published by Jeremy Howard of Answer.AI in 2024. The idea is simple: put a plain-text Markdown file at /llms.txt on your domain, the same way /robots.txt lives at the root. Where robots.txt tells crawlers what they may visit, llms.txt tells AI models and agents what is worth reading.
It's useful because AI agents have limited attention. A typical page is mostly navigation, scripts, tracking and layout. An agent asked \"what's the return policy at this store?\" has to dig through all of that. A good llms.txt hands it the answer's location in one request.
\nAn honest caveat: llms.txt is a proposal, not something every AI company has committed to read. Some coding assistants and documentation tools use it today, and it's cheap to add, but treat it as one part of agent readiness alongside clear pages, structured data and a sitemap, not a replacement for them.
\nThe file is ordinary Markdown with a light structure, in this order:
\n- [Page name](URL): what it's for.Here's an llms.txt for a fictional online store:
\n# Northwind Coffee\n> Northwind Coffee roasts and sells specialty coffee beans, ground coffee and\n> brewing equipment online, shipping to the US and Canada.\nOrders over $40 ship free in the US. We don't sell gift cards or wholesale online;\nwholesale enquiries go through the contact page.\n## Shop\n- [All coffee](https://northwind.example/collections/coffee): Every bean we sell, with roast level, origin and price\n- [Subscriptions](https://northwind.example/subscriptions): Recurring deliveries every 2, 4 or 6 weeks; pause or cancel any time\n- [Brewing equipment](https://northwind.example/collections/equipment): Grinders, kettles and pour-over kits\n## Help and policies\n- [Shipping](https://northwind.example/pages/shipping): Rates, delivery times and countries we ship to\n- [Returns](https://northwind.example/pages/returns): 30-day returns on unopened items and equipment\n- [FAQ](https://northwind.example/pages/faq): Grind sizes, freshness, subscriptions and account questions\n- [Contact](https://northwind.example/pages/contact): Email and live chat hours\n## Optional\n- [Our story](https://northwind.example/pages/about): Who we are and how we source our beans\n- [Brewing guides](https://northwind.example/blog/guides): Step-by-step guides for each brewing methodYou can also see our own llms.txt.
\nThink about the questions people ask agents about businesses like yours, and link to the page that answers each one.
\n| Type of site | Link to |
|---|---|
| Online store | Main categories, bestsellers, shipping, returns, sizing, FAQ, contact |
| Software company | Product overview, pricing, sign-up, docs, API reference, security, status page |
| Local or service business | Services and prices, booking, locations and hours, service area, contact |
| Publisher | Sections, subscription options, licensing and permissions, editorial policy |
If you offer agents a direct way in, such as an API or an MCP server, list it with a link to its documentation. AgentScore looks for an MCP server mentioned in llms.txt as one of the signals that a site supports agents directly.
\n/llms.txt in a browser and make sure you see plain text, not a web page.llms.txt.https://yourdomain.com/llms.txt. On most hosts this means putting it in the same folder as robots.txt. On Shopify, WordPress and other platforms, apps and plugins can serve it for you.Some sites, mostly documentation sites, also publish /llms-full.txt: the full text of their key pages in one Markdown file, so an agent can load everything at once. It's a community convention rather than part of the proposal. It's worth it for developer docs; for most stores and marketing sites a good llms.txt is enough.