Why run the scanner at scale

Every AgentScore check is tested against small, carefully built example sites. That proves the check does what we meant. It doesn't prove we meant the right thing for the thousands of ways real stores are built. Scanning 600 stores, from many countries, platforms and sizes, was the first time AgentScore met that variety all at once, and the first time we read its results in aggregate, where a check that's wrong for many stores stands out.

Bug 1: AgentScore only spoke English

Several checks work by reading what a button or link says: finding the add-to-cart button, recognising the button that closes a cookie banner, spotting the link to the cart. All of them only knew English words. On a German store, "In den Warenkorb" wasn't recognised as an add-to-cart button, and "Alle akzeptieren" didn't count as a way to close the banner.

In our first run, 61 stores whose pages weren't in English were reported as having no add-to-cart button at all. When we rescanned eight of them with the fix, AgentScore found the button on six. The other two turned out to be a different problem (see bug 3).

There was a subtler issue underneath. The usual way to match whole words in code only understands the 26 letters of the English alphabet, so even a correctly translated phrase could fail to match at the edges of words in Russian or with accented letters. Japanese, Chinese and Korean don't put spaces between words at all.

What we changed: the words AgentScore looks for now live in one list covering 19 languages, including German, French, Spanish, Italian, Dutch, the Nordic languages, Polish, Turkish, Russian, Ukrainian, Japanese, Chinese and Korean, and matching works in every alphabet. It also changed which sites AgentScore recognises as stores, because finding the cart link is part of that: the same sample produced 230 stores instead of 219.

Bug 2: Real buttons reported as fake ones

AgentScore checks that your add-to-cart button is a real button, because agents look for buttons and links, not for text that happens to be clickable. To find the button, it looked for the smallest element whose text matched. Many shop themes write their buttons like this:

<button type="submit">
  <span>Add to cart</span>
</button>

The smallest element containing "Add to cart" is the <span>, so AgentScore reported "a plain <span>, not a real button", on a perfectly good button. That's a false alarm, and an expensive one: in the study's second run, 57% of the stores where we found the button were flagged. After the fix, the true figure is 24%.

What we changed: when the matching text sits inside a real button or link, AgentScore now judges that button or link. A <div> styled to look like a button is still caught, because that really is a problem for agents.

Bug 3: Sold-out products counted against stores

AgentScore checks one product page per store. When that product was sold out, its buy button had been replaced by a disabled "Sold out" button, "売り切れ" on a Japanese store or "Agotado" on a Spanish one, and AgentScore reported that it couldn't find an add-to-cart button. That says nothing about the store's buttons, only about its stock.

What we changed: AgentScore now recognises a sold-out product, from its button in any of the 19 languages or from the product data saying it's out of stock. Then it does what a shopper would: it tries up to two other products. If they're all sold out, the check is marked as not tested, which never costs points. In the study, AgentScore moved past a sold-out product on 4 stores and skipped the check on 5 where everything it tried was sold out.

The run that looked perfect

One of our runs came back looking wonderful. Navigation scored 100%. Not one store had an unnamed button. No pop-ups blocked anything.

It was wrong. After an update to how AgentScore opens pages in its browser, the browser on our scanning machine couldn't complete secure connections, and it couldn't load a single page. Every check that needs a real browser was skipped, on 245 of 245 stores. Skipped checks don't cost points, by design, so the scores went up instead of down.

We caught it because the numbers were too good to be true, not because anything failed. That's not good enough, so we added a safeguard: our analysis now refuses to produce results if more than 10% of stores couldn't be opened in a browser. We fixed the scanning machine's setup, confirmed one scan by hand, and ran all 600 sites again.

What this means for your score

• Stores outside English-speaking markets get a fairer score, and some will see it rise, as buttons, banners and cart links are now recognised in their language.
• Stores whose buttons wrap their text in another element no longer get a false "not a real button" finding.
• A sold-out product no longer costs you points.

All three changes are in AgentScore's current scoring version, 2026.10-v5. If you scanned your site before, run AgentScore again to see your updated score.

What we took away

• Test against the real world, not only your examples. Each bug passed every test we had, because our test sites were built the way we expected sites to be built.
• Be suspicious of good news. The broken run would have made a better headline than the real one. A result that's surprisingly good deserves the same scrutiny as one that's surprisingly bad.
• Publish the limits. The study states what AgentScore can't see, such as checkouts that need items in the cart and requests that don't come from AI companies' verified addresses. Saying so is what makes the rest of the numbers worth trusting.

Curious how AgentScore opens and reads your pages? How AgentScore renders pages explains the browser side, and How AgentScore works covers the scoring.

How we did it

We drew 600 websites at random from the online shops listed in Wikidata, the public knowledge base, so the sample leans toward established retailers notable enough to have an entry, from many countries. On October 9, 2026 we ran the same automated AgentScore scan (scoring version 2026.10-v5) on each, and 544 could be scanned; the rest were unreachable, timed out or returned errors. 234 of those sites were identified as stores, with products or a cart, and those are the ones in this report. Another 12 stores showed a bot challenge even to an ordinary browser request from our cloud servers, so we couldn't see them at all and left them out.

For each store, AgentScore checks three things:

• Access: whether robots.txt rules, bot protection or CAPTCHAs stop AI assistants such as ChatGPT-User, Claude-User and Perplexity-User from reaching the home page, product pages and checkout.
• Readability: whether content, product details and prices are in the page's HTML, or only appear after JavaScript runs.
• Navigability: whether buttons, links and form fields have names agents can read, whether pop-ups block the page, and whether the add-to-cart button is a real, uncovered button.

This version doesn't send an AI agent through a live shopping task, so the scores cover whether agents can get in, read and find their way, not whether they complete a purchase. The scan is read-only: it never submits forms, adds to cart or places orders. All scans ran from a single cloud location, so pop-ups and content that vary by country may differ for visitors elsewhere. We report results only in aggregate and don't name individual stores.

The scores

The median store scored 85 out of 100. 65% scored 80 or more, and 1% scored below 50.

By category, the median store earned 92% of the available points for access, 94% for readability and 75% for navigability. Getting in and reading pages is mostly solved. Finding the way around is where agents struggle.

Finding 1: Most stores let AI assistants in

The good news: robots.txt rules blocked AI assistants or AI search on only 3% of stores. Counting every way we test, 14% of stores blocked or challenged AI assistants somewhere:

• 9% blocked or challenged a request that identified itself as an AI assistant, such as ChatGPT-User or Claude-User, on the home page, while the same request as a normal browser got straight through.
• 8% kept AI assistants or AI search out of product, pricing or cart pages, through bot protection or robots.txt.
• 5% had a CAPTCHA on the home page, which agents can't solve.

One caution on the bot protection figures: our requests used the AI assistants' names but came from our own servers, not from the operators' verified addresses. A site that checks where AI agents really come from would rightly turn some of these away, so these are upper bounds on how often real assistants are blocked. The robots.txt figure reads the rules each site publishes, so it isn't affected.

For the stores that do block assistants, the fix is usually a settings change, not a rebuild. See bot protection and CAPTCHAs and robots.txt for AI agents.

Finding 2: Product details are where readability slips

Most AI crawlers and assistant fetchers read the HTML a server sends and never run JavaScript. Here most stores do well: only 6% had less than 70% of their home page text in the HTML.

Product pages are a different story:

• 63% of stores give price, currency and availability as structured data in the product page's HTML, the way agents read most reliably.
• 24% have no usable product structured data at all, so agents have to guess price and stock from the layout.
• 6% have product data that's incomplete or only in older microdata, and 7% only add it with JavaScript.
• 30% don't have prices in the product page's HTML at all.

An agent that can't see a price can't compare you, and an agent that can't tell whether something is in stock will usually recommend a store where it can. See product data AI shopping agents can read.

Finding 3: Navigation is the weakest link

Navigability was the lowest-scoring category, and the most common problems in the whole study are here:

• 48% of stores have buttons or links with no name an agent can read, most often icon-only cart, search and menu buttons.
• 47% have elements that look clickable but aren't real buttons or links.
• 39% show a pop-up or banner that covers much of the page on arrival. On 28% of stores, it had no clearly labelled button to close it.
• Of 118 stores where we found the add-to-cart button, 24% had one that isn't a real button, or is covered by something else, so an agent's click may not land.

These are the same problems that trip up people using screen readers, and fixing them helps both. See buttons, links and forms agents can use.

Finding 4: Checkouts mostly let agents through

AgentScore also loads each store's cart and checkout pages, without adding anything to the cart. Of 200 stores whose cart or checkout we could load, 8% blocked or challenged requests identifying as AI assistants there while browsers got through, and another 14% showed a CAPTCHA to every visitor. The same caution about verified addresses applies.

Most checkouts only show their first step once something is in the cart, so we could inspect too few of them to report on guest checkout or checkout forms. See how to make checkout work for AI shopping agents for what to look for in your own.

Finding 5: More stores speak to agents than we expected

Some stores now offer agents a direct way in. 53% publish an llms.txt file, 32% offer an MCP server or WebMCP tools, and 29% publish an agentic checkout profile. These standards are young, so we expected far lower numbers. They suggest agent-ready commerce is arriving faster than most store owners realize.

The most common problems

Across every check that covered at least 50 stores, these fell short most often (a fail or a warning):

What the best stores do differently

We compared the top quarter of stores by AgentScore with the bottom quarter. These checks showed the biggest gap in pass rates:

The leaders describe their products in a form agents can read, put prices in the HTML, publish an llms.txt file and offer agents a direct way in. Several of these are a one-time template or settings change, which makes them the cheapest points available to most stores.

What to fix first

See where your store stands

Run AgentScore on your own site. It's free, takes about two minutes, and shows exactly which of these problems apply to you, with a fix for each. To understand the bigger picture, start with What is agent readiness?

Method notes: scores use AgentScore scoring version 2026.10-v5. Percentages use the stores where each check could run; checks that didn't apply to a store, or couldn't be tested, are left out of that check's total. Scans ran on October 9, 2026, and sites change, so a store's score today may differ. Checks that read what buttons and links say, such as "Add to cart" or "Close", understand 19 languages, so stores in every language count toward every figure.

The page

Northwind Outfitters sells running gear at northwind.example. Its product page for the Trail Runner 2 looks good to a person: a large photo, the name, a price of $129, a row of size swatches, an "Add to cart" button and, on a first visit, a cookie banner across the bottom half of the screen. Like many modern stores, the page is built in the browser: the server sends a shell, and JavaScript fetches the product and fills it in.

A shopper asks an AI assistant: "Find me a trail running shoe under $150 in a size 10 that I can get this week." Here is what Northwind's page looks like to the agents that might come to answer.

View 1: the raw HTML

AI assistants that fetch a page because someone just asked a question, and the crawlers behind AI search, usually request the page and read the HTML that comes back. Many don't run JavaScript at all. This is what Northwind's server sends:

<html>
<head>
  <title>Northwind Outfitters</title>
</head>
<body>
  <header><a href="/"><img src="/logo.svg"></a></header>
  <div id="app">Loading…</div>
  <script src="/assets/app.4f9c.js"></script>
</body>
</html>

That's the whole story for this kind of agent. No product name, because the title is the store's name. No price, no sizes, no stock and no structured data. The assistant can't tell the shopper the Trail Runner 2 costs $129 or comes in a size 10, so it recommends a shoe from a store whose page it could read.

Nothing on screen warns Northwind's team about this. The page looks perfect in every browser they own. It's the problem our guide to JavaScript-only content is about, and the reason Product structured data and machine-readable prices belong in the HTML the server sends.

View 2: the rendered page

Browser agents run a real browser. They load the page, let the scripts run, and decide what to do next, often by looking at a screenshot. This agent gets much more:

• The photo, the name "Trail Runner 2" and the price, $129.
• The size swatches, though nothing on screen says which sizes are in stock: sold-out sizes are only a lighter shade of gray.
• The cookie banner, covering the bottom of the screen, including the "Add to cart" button.

The agent has to deal with the banner first. If its buttons are clearly labelled "Accept" and "Reject", that's one extra step. If the only way out is a small "×" drawn in an image, the agent may click the wrong thing or give up. Either way, it costs a step on every visit where the agent starts with a fresh browser, and many do. Our guide to cookie banners and pop-ups covers how to keep them out of the way.

View 3: the accessibility tree

Many browser agents don't act on pixels alone. They read the page's accessibility tree: the same outline of headings, buttons, links and fields that screen readers use, with a role and a name for each item. It's compact, and it tells the agent exactly what it can press. Here is a simplified version of Northwind's:

banner
  link (no name)
  button (no name)
main
  heading "Trail Runner 2" level 1
  text "$129.00"
  generic "8"
  generic "9"
  generic "10"
  generic "11"
  button "Add to cart"
dialog "We value your privacy"
  button "Accept all"
  button (no name)

Read it the way an agent would:

• The logo link and the cart icon have no names. The agent can't tell which button is the cart. An aria-label or visible text fixes both.
• The sizes aren't controls. They're plain boxes with a click handler, so they show up as "generic" rather than buttons or options. The agent may not realize it can choose a size at all, and nothing tells it which sizes are sold out. Real buttons or radio inputs, with the stock state in their names, solve it. See variant pickers AI agents can use.
• The banner's "Reject" button has no name. It's an icon with no label. An agent that wants to decline cookies on the shopper's behalf can't find the option.

The good news is in there too: one clear h1, a price as text, and an "Add to cart" button that is a real button with a real name. Those are the things that let an agent finish the job.

Same page, three stories

No single view is "the" agent view. A shopper's question might be answered from the raw HTML, and the purchase made by a browser agent working from the accessibility tree. A page has to work in all three.

How to see your own pages this way

1. Raw HTML: open a product page, choose "View page source", and search for the price and the product name. If they're not there, agents that read the HTML don't have them.
2. Rendered page: open the same page in a private window at desktop size. What covers it on arrival is what a browser agent meets first.
3. Accessibility tree: in Chrome's developer tools, the Accessibility pane shows each element's role and name. Look for buttons and links with no name and for clickable things that show up as "generic".

AgentScore does this comparison for you. It fetches your home page and key pages as a browser and as AI assistants, renders them in a real browser, and checks names, labels, banners and the add-to-cart button. Checks such as Content loads without JavaScript, Product pages give price and stock in a form agents can read, Buttons and links have names and Pop-ups and banners can be dismissed by agents map directly onto the problems above. How AgentScore renders your pages explains the details.

Why peak season is different

Every change made for peak is reasonable on its own. Security tightens bot protection to stop scalpers and card testing. Marketing adds a countdown banner and an email pop-up. Merchandising loads sale prices through a promotions app. Operations shortens cache times, or lengthens them to cope with load. Then engineering freezes the code.

Each of those changes is made by a different team, often in a hurry, and none of them is tested with an AI agent in mind. An assistant asked to "find a gift under $50 that arrives before the 24th" needs to get past your bot rules, read the sale price, see that the item is in stock, close the pop-up and reach the cart. Any one of those can fail quietly, and you won't see it in your analytics.

A timeline for the weeks before the freeze

1. Bot protection tightened for peak

This is the change most likely to turn agents away. Common peak settings include stricter challenge thresholds, "under attack" modes that challenge every visitor, blocks on cloud and data center networks, and country blocks. Browser agents and assistants that fetch pages for people often run from cloud networks, so they get caught by rules written for scrapers.

• Ask for a list of every bot rule planned for peak, and who can change them during the freeze. CDN rules often sit outside the code freeze and change mid-season.
• Make sure the AI assistants you want are allowed by verified identity, not just by user agent. See how to tell if an AI crawler is real.
• Check cart and checkout separately from the home page. Many stores add extra protection there for peak.
• Prefer rate limits on the expensive actions, like payment attempts, over blanket challenges. See rate limits for AI agents and geo-blocking and AI agents.

AgentScore checks whether your bot protection treats AI assistants differently from a browser, and whether the cart and checkout show them a block or a CAPTCHA. Rescan after every bot rule change, not just once in October. For the settings themselves, see bot protection and CAPTCHAs and setting up your CDN for AI agents.

2. Sale prices that agents can read

Promotion tools often change the price in the browser with JavaScript after the page loads. A shopper sees the sale price. An agent that reads the HTML sees the full price, or no price at all, and may quote the wrong figure to its user or skip you.

Before peak, run a test promotion and check three places: the visible price in the page source (use View Source, not the inspector), the price in your product structured data, and the price in the cart. They should all agree. If a sale ends on a known date, say so:

"offers": {
  "@type": "Offer",
  "price": "39.00",
  "priceCurrency": "USD",
  "priceValidUntil": "2026-12-01",
  "availability": "https://schema.org/InStock"
}

Show the original and the sale price in text, such as "Was $59, now $39", rather than relying on a strikethrough alone. See prices AI agents can read.

3. Stock that stays accurate

At peak, stock changes by the hour. If your structured data says InStock after an item sells out, an assistant will send people to a product they can't buy. If it says OutOfStock for an item you restocked this morning, you lose the recommendation.

• Make sure availability in your product data comes from the same source as the "Add to cart" button, not a nightly export.
• Use PreOrder or BackOrder where they apply, and state delivery estimates in text.
• If you lengthen CDN cache times for load, check how stale product pages can get, and purge on stock changes for best sellers.

4. Promo pop-ups and banners

Email capture, spin-to-win wheels, countdown banners and free-shipping bars all multiply in November. An agent can only get past one if it has a real, labelled close button and doesn't cover the content behind it. A pop-up that sits over the "Add to cart" button means the agent's click lands on the pop-up instead.

AgentScore checks that pop-ups and banners can be dismissed, and whether the add-to-cart button on a product page is covered by something else. Check again once peak promotions are live, because they're usually added after the last scan. See cookie banners and pop-ups.

5. Promo codes, carts and checkout

Gift shopping by agent depends on the last few steps working. Make sure the promo code field is labelled, that a rejected code produces a clear error in text, and that the cart total updates in the page. See carts AI agents can manage and error messages AI agents can understand.

Resist adding a CAPTCHA to checkout for peak. It stops agents at the very last step, after they've done all the work. Card testing is better handled by your payment provider's fraud tools and by rate limits on payment attempts. Keep guest checkout on. See checkout for AI shopping agents and guest checkout.

6. Holiday policies in plain text

Assistants are often asked about delivery cut-off dates and extended returns. Put them on your shipping and returns pages as text in the HTML, with dates, not only in a banner image or a pop-up. See policy pages AI assistants can read.

Planning for the freeze itself

A code freeze protects you from risky changes, but it also means a problem found in December may wait until January. Three things help:

1. Agree that blocking AI agents counts as an incident. If an assistant can't reach your product pages or checkout, that should qualify for a freeze exception, the same as a broken payment form.
2. Test the money journeys on a schedule. Ghost Agents run real journeys, such as search, choose a size and add to cart, stop at the payment form, and alert you when a journey that used to pass starts failing, with a step-by-step replay.
3. Watch your logs, not just analytics. Most agents don't run your analytics tag. A rise in blocked or failed agent requests shows up in server and CDN logs first. See finding the errors AI agents hit.

Start with a fresh AgentScore scan, then work through the list. If you have more time, our 90-day agent readiness plan covers the rest.

How AgentScore sees a software site

AgentScore looks for the pages an agent acting for a customer would need, the way an agent would: links in your home page's HTML, then your sitemap, then common addresses such as /pricing. If it finds product pages or a cart, it treats the site as a store. If it finds a pricing page and no products or cart, it treats the site as software ("saas" in the scan's data). That changes what some checks look for.

Everything else, from robots.txt and bot protection to structured data, named buttons and labelled form fields, applies the same way as for any site. See what is agent readiness? for the full model.

Pricing pages agents can read

Pricing pages are often the most designed page on a software site, and that's where agents struggle. Common problems:

• Prices loaded by JavaScript. A monthly/annual toggle that fetches prices when clicked leaves the HTML with no prices at all. Put both sets of prices in the HTML and let the toggle show and hide them.
• Features shown only as icons. A comparison table of tick and cross images says nothing to an agent unless each icon has a text label, such as "Included" or "Not included". See image alt text.
• Units left implicit. Say "per user per month, billed annually" in text, not in a tooltip. Agents compare plans across vendors, and unclear units make you look more expensive or less clear than a competitor.
• "Contact sales" with no explanation. If a plan is priced on request, say so in plain text and say what affects the price. An agent can then tell its user how to get a quote instead of guessing.

A plain HTML table is still the clearest way to compare plans:

<table>
  <caption>Northwind plans, billed annually</caption>
  <tr><th></th><th>Starter</th><th>Pro</th></tr>
  <tr><th>Price</th><td>$12 per user per month</td><td>$29 per user per month</td></tr>
  <tr><th>Single sign-on</th><td>Not included</td><td>Included</td></tr>
</table>

For more, see prices AI agents can read.

Sign-up and trials

An agent that has chosen your product for someone will try to start the trial. It needs a visible, real button with a clear label, then a form it can fill in: every field labelled, errors explained in text, and no surprises. See buttons, links and forms agents can use.

Some steps should stay with the person, and that's fine. Email verification, single sign-on, payment details for a card-required trial and accepting terms are all reasonable places for an agent to hand back to its user. The aim is for the agent to get as far as it should, then stop cleanly with a clear message, rather than fail at a CAPTCHA on the first screen. See sign-up, booking and lead forms AI agents can finish.

Ghost Agents can test this journey for real: from the home page to pricing to the sign-up form, using test data you supply, with a step-by-step replay showing where an agent got stuck.

Docs are your product page for agents

For many software products, the docs decide the sale. Coding assistants and research agents read them to answer "does it support X?" and "how hard is it to set up?". Treat docs with the same care as marketing pages:

• Serve docs as HTML that works without JavaScript. Some docs sites are single-page apps that send an empty shell. See JavaScript-only content.
• Keep public docs public. If setup guides sit behind a login, agents can't use them to recommend you. See login walls and gated content.
• Give each version a stable address and point old copies at the current one, so agents don't quote outdated instructions. See canonical URLs.
• Publish limits, supported integrations and security information as text, not only in PDFs or sales decks.

llms.txt for software companies

llms.txt is a proposed convention, not a standard, and AI providers don't all say whether they read it. It's cheap to publish, though, and it suits software companies well, because your most useful pages are easy to list. Some docs platforms can generate one for you.

# Northwind
> Northwind is scheduling software for field service teams.
## Product
- [Pricing](https://northwind.example/pricing): plans, limits and what each includes
- [Start a free trial](https://northwind.example/signup): 14 days, no card required
## Docs
- [Getting started](https://northwind.example/docs/start)
- [API reference](https://northwind.example/docs/api)
- [Integrations](https://northwind.example/integrations)
## Trust
- [Security](https://northwind.example/security)
- [Status](https://status.northwind.example/)

AgentScore checks that an llms.txt file exists. It can't tell whether yours is up to date, so add it to your release checklist. See how to write an llms.txt file.

MCP: your product, not just your website

For a software company, the Model Context Protocol is about more than your website. An MCP server can let an assistant use your product itself on a customer's behalf: look up records, create tasks, run reports. That's product work, with sign-in and permissions to design carefully, and many software companies are already building one.

There's no single agreed way to advertise an MCP server on a website yet. AgentScore's "Agents can use your site through MCP or WebMCP" check looks for a server card at /.well-known/mcp.json, an MCP link in the page, a mention in llms.txt, or WebMCP tools in the page. If you have a server, mentioning it in llms.txt and your docs is the simplest start. See MCP and WebMCP for websites.

Where to start

1. Run AgentScore on your home page. If "Key pages are linked from the home page" doesn't pass, agents may not be finding your pricing page.
2. View the source of your pricing page and check every price and plan feature is there as text.
3. Make "Start free trial" a real, clearly labelled button, and check the sign-up form's fields are labelled.
4. Publish an llms.txt that links to pricing, sign-up, docs and your trust pages.
5. Test the journey from home page to sign-up with a Ghost Agent.

The one-page version

Start with this structure. Each heading is one or two sentences. Fill in the bracketed parts with your own findings.

That's it. Everything below is about filling it in well.

Explaining the shift without hype

Boards have heard a lot about AI. The quickest way to lose the room is a slide of large market forecasts. You don't need them. The argument for agent readiness is modest and easy to defend:

• AI agents already visit websites for people. You can show this from your own server or CDN logs.
• Nobody knows how fast this will grow. That's a reason to be ready cheaply now, not to wait for certainty.
• Much of the work also improves accessibility, search and site quality, so it isn't a bet on one trend. See accessibility work is agent readiness work and SEO got you found. Agent readiness gets you chosen.
• The failure is silent. A blocked agent doesn't complain. It goes elsewhere. That makes it the kind of risk a board should want someone to own.

If you quote any outside figure, cite the source on the slide. If you can't, describe the trend in words instead.

The risks, in board language

The point of the table is that this is a cross-team issue with no natural home. Ask the board to agree one accountable owner. See who owns agent readiness?

What to measure

Pick a few measures you can report every quarter, and show the trend rather than a single figure.

• AgentScore, by category. Access, Readability and Navigability tell you whether agents can get in, understand your pages and find their way around. The free scan doesn't test Task completion, so don't present its score as proof that agents can buy.
• Journey pass rate. Whether AI agents can complete your most important journeys, such as checkout or sign-up, from scheduled Ghost Agent tests.
• Agent traffic. How many requests come from AI assistants and search agents, how many are verified, and how many are impostors, from your logs. See how to measure agent traffic.
• Visits and orders from AI assistants. People who click through from an assistant's answer. See tracking visits and sales from AI assistants.
• Time to fix. How long a critical finding, such as AI agents being blocked, stays open.

Be careful with attribution. You can show that more agents get through and more journeys pass. You usually can't prove exactly how much revenue that produced, and the board will trust you more if you say so. For a fuller set, see agent readiness KPIs.

What to ask for

Keep the ask small and staged, with a review point after each stage. A typical shape:

1. A baseline. A scan, a look at agent traffic in your logs, and one tested journey. This needs a few days of people's time, not a project.
2. Fixes. Developer and content time to work through the findings. Much of it overlaps with work already on the accessibility and SEO backlog, so say where it does.
3. Ongoing testing and monitoring. Tools that keep watching after the fixes, because a theme update or a new bot rule can undo them.
4. Decisions. Time from legal and security to settle policy questions such as AI training crawlers and verified agents.
5. Optional experiments. Newer interfaces such as MCP and WebMCP or agentic commerce protocols, framed as small trials with a date to decide whether to continue.

Our 90-day agent readiness plan maps well onto the first three stages.

Questions you'll probably get

Are we letting AI companies take our content?

No. AI training crawlers and the assistants that fetch pages for a customer use different names, so you can block one and allow the other. That's a policy choice to make once, with legal. See should you block AI crawlers?

Does this open us up to bots?

Not if it's done properly. The goal is to let in the agents you want, checked by verified identity, and keep blocking the rest. See how to tell if an AI crawler is real.

What happens if we do nothing?

Probably nothing you'll notice, which is the problem. Some customers' assistants will fail on the site and choose a competitor, and it won't show up in any report you currently read.

To fill in "Where we stand", run a free AgentScore scan. It takes under a minute and gives you a score out of 100 and a list of what to fix.

1. Blocking AI assistants along with training crawlers

Many sites decide not to let AI companies train on their content, which is a fair choice. The mistake is blocking everything with "AI" in its name, including the assistants that fetch a page because a customer asked a question right now. Those are different agents with different names: GPTBot collects training data, while ChatGPT-User and OAI-SearchBot fetch pages for ChatGPT users and search. Anthropic draws the same line between ClaudeBot and Claude-User.

# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
# Everyone else, including AI assistants and AI search
User-agent: *
Allow: /

Instead: decide on training and on assistants separately. See should you block AI crawlers? and robots.txt for AI agents. The same split applies to bot protection rules, which often block by category rather than by name.

2. Putting a CAPTCHA at checkout

A CAPTCHA at checkout is often added to stop card testing, and it works on bots. It also stops every AI agent at the very last step, after it has searched, compared, chosen a size and filled the cart. That's the most expensive place to lose a customer.

Instead: use your payment provider's fraud tools and rate limits on payment attempts, and keep challenges for traffic that is actually suspicious. AgentScore checks whether AI agents are blocked or shown a CAPTCHA at the cart and checkout. See checkout for AI shopping agents and bot protection and CAPTCHAs.

3. Prices that only appear with JavaScript

Many sites build product and pricing pages in the browser. A person sees the price a moment after the page loads. Many AI agents read the HTML your server sends and never run the scripts, so they see a product with no price, or a placeholder. They may skip you, or tell their user to check the site.

Instead: make sure prices are in the HTML, through server-side rendering or static generation, and in your product structured data. Check by viewing the page source, not the browser's inspector, which shows the page after scripts run. See prices AI agents can read and JavaScript-only content.

4. Icons with no names

A magnifying glass for search, a bag for the cart, a cross to close a pop-up. People know what they mean. Browser agents choose what to click by each control's name, the same way screen readers do, and an icon with no text or label has no name at all. To the agent, the cart button isn't there.

<button aria-label="Open cart">
  <svg aria-hidden="true">…</svg>
</button>

Instead: give every icon-only button and link a label, and use real <button> and <a> elements rather than clickable boxes. See buttons, links and forms agents can use and accessibility work is agent readiness work.

5. Publishing llms.txt and forgetting it

llms.txt is a proposed convention for giving AI a short guide to your site. It's quick to write, which is also why it goes stale: a file written at launch still links to last year's collections, a retired plan, or a returns page that moved. An out-of-date guide can be worse than none, because it points agents at the wrong answer.

Instead: give llms.txt an owner and add it to the checklist for site changes, alongside the sitemap. Note that AgentScore checks that the file exists, not whether what it says is still true, so that part is up to you. See how to write an llms.txt file.

6. Trusting user agents

A user agent is a name a visitor gives itself, and anyone can use any name. If you allow a list of AI agents by user agent alone, scrapers borrow those names to get in. If you block by user agent, you may turn away a real assistant while impostors simply change their name.

Instead: verify identity. The large operators publish their IP ranges, and many crawlers can be confirmed with a reverse DNS lookup. Newer agents can sign their requests with Web Bot Auth, which your CDN may already check for you. See how to tell if an AI crawler is real and why we sign every request our agents send. Ghost Agent Labs' Verification page checks every request that claims to be a known agent and shows the impostors.

7. Measuring AI agents with JavaScript analytics

Analytics tools such as Google Analytics count visitors by running a script in the browser. Most AI agents don't run it, so they never appear. A report that shows no AI agents isn't evidence that none came. It may mean they came and the tag didn't see them, or that they were blocked before the page loaded.

Instead: measure agents from your server or CDN logs, which record every request whether or not scripts run. Keep analytics for the people who click through from an assistant's answer, which it can see. See why your analytics can't see AI agents, AI referral traffic and agent traffic isn't bot traffic.

What these have in common

None of these mistakes shows up from inside the business. The site looks fine in a browser, the dashboards look normal, and the security team sees fewer bad bots. The cost lands on customers using an assistant, who quietly go elsewhere.

A free AgentScore scan catches several of these in under a minute. For the journeys themselves, Ghost Agents show whether an agent can actually finish them.

Why render at all

Several checks only make sense as a comparison between the page before and after JavaScript. Content loads without JavaScript measures how much of the rendered page's text was already in the HTML: 70% or more passes, 30% to 70% is a warning, and less than that fails. Structured data describes your business and products, Product pages give price and stock in a form agents can read and Prices are in the page HTML all look for the same thing: data that only appears once scripts have run, which agents that read the HTML never see. Our guide to JavaScript-only content covers why that matters.

The Navigability checks need a rendered page too: whether a button is visible or a banner covers the screen depends on layout, which only exists once a browser has drawn the page.

Which pages we render

A scan renders at most four pages, each in its own browser session:

The cart is fetched but not rendered. How AgentScore works explains how key pages are found and lists every page a scan requests.

The browser and its settings

• Chromium, headless. The engine behind Chrome, driven by Playwright.
• A fresh, isolated session for every page. No cookies or storage carry over, so every page sees a first-time visitor.
• A desktop screen, 1280 by 900 pixels, and a desktop Chrome user agent. These requests aren't signed, so your site treats them as it would any desktop visitor.
• No downloads and no service workers.
• Up to 20 seconds to load. We wait for the HTML to be parsed, then up to 5 more seconds for network activity to settle, so content built in the browser and consent banners have time to appear. Pages with chat widgets or analytics beacons rarely go quiet, so after 5 seconds we inspect whatever is there.

If a page fails to load or render, the checks that depend on it are marked "not tested" with a note explaining why. A render failure never counts as a fail.

What we inspect once it's drawn

On the home page, a script reads the finished page and records:

• Every visible button and link, and whether it has a name an agent can read: its text, an aria-label, a referenced label, a title, or the alt text of an image inside it.
• Every visible form field, and whether a label is connected to it. Fields with only placeholder text are counted separately, as a warning.
• The largest fixed or sticky element covering at least 15% of the screen, leaving out ordinary header bars along the top, and whether it has a clearly labelled button such as "Accept", "Close" or "No thanks".
• Things that look clickable but aren't buttons or links, the <main> and <nav> landmarks, and images with no alt attribute.

On product and pricing pages, we look for the main action by its text, such as "Add to cart", "Buy now", "Start free trial" or "Sign up". We scroll it into view, check whether it's a real button or link, and ask the browser what element sits at its center. If that's a cookie banner rather than the button, an agent's click would land on the banner, and Agents can find and press your add-to-cart or sign-up button fails.

The WebMCP probe

WebMCP is a proposal for a browser feature, navigator.modelContext, that lets a page offer tools directly to an AI agent in the browser, such as "search products" or "add to cart". It isn't generally available in browsers yet, so sites that support it check for it first, roughly like this:

if ("modelContext" in navigator) {
  navigator.modelContext.registerTool({
    name: "search_products",
    description: "Search the Northwind catalog by keyword",
    inputSchema: { type: "object", properties: { query: { type: "string" } } },
    execute: async ({ query }) => searchCatalog(query),
  });
}

Before your scripts run, we add a stand-in for navigator.modelContext that writes down the name of every tool a page registers. A page that checks for WebMCP then registers its tools just as it would in a browser that supports it. We also look for forms marked up with a toolname attribute. We run the probe on the home, product and pricing pages and record names only: our stand-in never calls a tool. The result feeds Agents can use your site through MCP or WebMCP, where a missing setup is a warning, never a fail. Our guide to MCP and WebMCP for websites explains both.

Keeping the browser safe

A service that opens any URL it's given is a target: someone could submit an address that points at an internal network or a cloud metadata service. So every connection the scanner makes is checked:

1. Only http and https. Other schemes are refused.
2. Public addresses only. Every address a host name resolves to must be publicly routable. Private, loopback, link-local, shared, reserved, documentation and multicast ranges are refused, for IPv4 and IPv6, including IPv4 addresses hidden inside IPv6 ones. Cloud metadata addresses are always refused.
3. Checked twice. Each request the page makes is checked before it goes out. Then all browser traffic, WebSockets included, passes through a small proxy that looks up the host name itself and connects only to addresses that pass. A host can't answer with a public address for the check and a private one for the connection.
4. Limits on plain fetches. Requests made without the browser are checked again at every redirect, follow at most five redirects, time out after 15 seconds and stop reading at a size limit, 5 MB for an ordinary page.

What the browser never does

• It never clicks, types, signs in, adds to cart or submits a form. The only interaction is scrolling the main button into view to see what's on top of it.
• It doesn't dismiss your cookie banner. We measure the page as a first-time visitor meets it, because that's how most agents arrive.
• It doesn't pretend to be an AI agent. Comparisons between agents use plain requests, as described in Why our scanner sometimes wears other agents' names.
• It doesn't test a mobile layout, and it can't see a checkout that only opens with items in the cart. Checks that depend on one say "not tested" rather than guess.

Finishing a real task is a different job. That's what Ghost Agents do in the app: real AI agents run your journeys, stop at the payment form, and give you a step-by-step replay. The free scan marks Task completion as not tested and scores the other three categories.

Start at the top: score, band and verdict

The score is out of 100. It falls into one of four bands, and the verdict under it says what the band means:

If any check failed, the verdict ends with Biggest issue: the summary of the failed check that carries the most weight. It's the single best place to start. If nothing failed, there's no biggest issue, and your work is in the warnings.

One thing to know before you share the number: the free scan doesn't run an AI agent through a task, so Task completion shows as "n/a" and the score is worked out over Access, Readability and Navigability. A 72 means 72% of the points that could be tested. How AgentScore works has the arithmetic.

Then the categories

Each category gets a bar and its points, such as Access 20/25. The app shows the same thing as a share of the category's points, so Access 20/25 reads as 80/100. Use the bars to see where points are being lost, not to rank your work: a low Navigability bar with one failure in it can matter less than a single Access failure.

Access comes first for a reason. If AI agents are blocked by your bot protection or robots.txt, they never see the pages the other two categories are about.

What each status means

Checks are listed failures first, then warnings, then passes, then not tested. Each one has a one-line summary of what was found, often with an example, such as how many buttons have no name or which assistant was blocked.

Don't skip "not tested"

A "not tested" check never costs you points, and its summary always says why it was skipped. The reasons fall into a few groups:

• It doesn't apply. Guest checkout, checkout fields, product data and agentic commerce are for stores. If the scan found no products or cart, they're skipped.
• It needs a full cart. The scanner never adds anything to a cart, so a checkout that only opens with items in it can't be inspected.
• A page couldn't be rendered. If the browser couldn't load a page, the checks that depend on it are skipped and say so. If it happens on every scan, ask a developer to check whether that page loads for a first-time desktop visitor.
• It isn't in the free scan. "An AI agent completes a real task" is always not tested there.

Read these anyway. If you run a store and the report says no product page was found from the home page or sitemap, that's worth knowing in itself: agents look for products the same way the scanner does.

Fixes, severity and score impact

Every warning and failure comes with a fix: the specific change to make. In the free report, you see the findings straight away and unlock the fixes with your email address. In the app, every fix is shown.

The app's Findings page adds three things that make the list easier to work through:

• Severity. A failure on a check with weight 4 or more is Critical, weight 2 or 3 is High, and weight 1 is Medium. A warning is Medium on a check with weight 3 or more, otherwise Low. A missing llms.txt is labelled Opportunity: worth doing, but a new standard, weighted lightly.
• Score impact. How many points your overall score would gain if that finding were fixed. On a store where every Access check was tested, a failed Bot protection lets AI agents through is worth about 7 points; a missing llms.txt is worth about 1. The numbers depend on which checks could be tested on your site, so read them from your own report.
• Why it matters and who usually fixes it. Open "Why it matters and how to fix it" under any finding for a plain-English reason, the fix, and the role that usually owns it: Developer, Bot protection or CDN admin, SEO or content team, or E-commerce platform admin.

The Readiness page shows the top four as Biggest wins. The Findings page lets you filter by severity, and Export CSV downloads every finding with its severity, score impact, summary, why it matters, fix, who usually fixes it and the scan date, ready for a ticketing tool.

How to decide what to fix first

1. Fix Access failures first. Blocked assistants, challenges on arrival and CAPTCHAs at the checkout stop agents before anything else matters. These usually sit with whoever runs your CDN or bot protection, and are often a settings change rather than a project. Our guide to bot protection and AI agents covers the common causes.
2. Then the remaining Critical and High findings, by score impact. Content that only appears after JavaScript, a banner covering the add-to-cart button, and product pages with no price data are typical.
3. Group by owner, not by category. Sort the CSV by "Usually fixed by" and send each person their own short list. One ticket per owner moves faster than one long list for everyone.
4. Pick up cheap warnings along the way. Image alt text, page landmarks and a missing meta description are small changes that often ship with other work.
5. Leave opportunities until the basics pass. llms.txt, MCP and agentic commerce are worth doing, and score lightly for a reason: they help most once agents can already get in and read your pages.
6. Rescan after each batch. In the app you can rescan whenever you like and watch the trend. The free page reuses a domain's result for 24 hours.

What the report can't tell you

A high score means your site removes the common obstacles. It doesn't prove that an AI agent can find a product, choose a size and reach the payment step. That's what Ghost Agents test in the app: real AI agents run your key journeys, stop at the payment form, and give you a step-by-step replay of where they got stuck. Use the report to clear the path, and journey tests to prove it's clear.

The two-pile habit

The people-or-bots split made sense for a long time. Apart from search engine crawlers, which everyone learned to welcome, automated traffic was mostly scrapers, credential stuffers, inventory hoarders and uptime monitors. So the tools grew up around it. Bot protection scores each request on how human it looks. Analytics filters out known bots. Dashboards show "bot traffic" as one line, usually as a problem.

AI agents arrive in that system looking like bots, because technically they are. But "bot" now covers software doing very different jobs for very different reasons, and some of those jobs are done for a specific person who wants to buy something.

Five kinds of automated visitor

Google-Extended is a robots.txt token rather than a separate crawler, but it controls the same choice. Running through all five rows is a sixth problem: impostors. Scrapers often claim to be Googlebot or GPTBot to get past bot protection, so a name in a user agent proves nothing on its own.

The top two rows are the ones the two-pile habit hurts most. An assistant fetch is one person's question. If it's blocked, that person gets an answer about somebody else. A browser agent that hits a CAPTCHA at checkout is an abandoned cart with no record of why. Neither shows up in your analytics, because most agents never run the tracking script, and both look like "bots" to a system tuned to stop bots.

What goes wrong when it's all one pile

• Blocking the wrong visitors. A rule written to stop scrapers stops ChatGPT-User too, and nobody notices because nobody is measuring it. Our guide to bot protection and AI agents covers how this happens.
• Making the wrong call on "AI". Blocking training is a reasonable choice. Blocking every AI user agent to achieve it also removes you from assistants and AI search. Should you block AI crawlers? splits the decision properly.
• Reading growth as a threat. "Bot traffic is up" sounds like an attack. "Assistant fetches of our product pages are up" sounds like demand. They can be the same line on the same chart.
• Missing the failures. If agent traffic is filtered out of reporting, so are the 403s, challenge pages and missing pages agents keep hitting. The journey breaks and the dashboard stays green.

Treat agent traffic as its own channel

The fix isn't to welcome every bot. It's to stop treating them as one thing. In practice that means four habits:

1. Measure it separately. Your server or CDN logs see every request, including the ones that never run JavaScript. Split them into the kinds above. How to measure agent traffic shows how.
2. Verify before you trust. Check a claimed agent against its operator's published IP ranges or reverse DNS, or a request signature where the operator signs its requests with Web Bot Auth. Our guides to verifying AI crawlers and signed requests cover the methods.
3. Set rules per kind, not per "bot". Let verified assistants and AI search through to product, pricing, cart and checkout pages. Decide on training crawlers as a separate business question. Rate-limit and challenge the rest, and block impostors outright. Rate limits for AI agents covers the middle ground.
4. Watch outcomes, not just volume. For each kind, track what it got back: pages served, redirects, blocks and missing pages. A rise in blocked assistant requests is a lost-sales signal, and should reach the same people as a broken checkout.

What it means for your reports

The most useful change is often the simplest: stop putting AI agents in the "bots" line. Give assistants, browser agents, AI search and training crawlers their own lines, alongside people, and report what each one got back. When a leader asks "is AI a threat or an opportunity for us?", that report answers with your own numbers instead of an opinion. Our guide to agent readiness KPIs suggests what to put in front of them each month, and tracking AI referral traffic covers the visits and sales that follow.

In the Ghost Agent Labs app, Agent traffic does this split from your server or CDN logs: people, search crawlers, AI training crawlers, AI assistants and autonomous agents, with each assistant's fetches grouped into sessions and a breakdown of how often agents were served, redirected, blocked or sent to missing pages. Verification separates verified agents from impostors.

Your next customer may arrive as a request with a user agent you've never looked at. It's worth knowing which pile you've put it in.

What SEO was built for

Search engines crawl, index and rank. A person sees a list of results, clicks one, and does the rest themselves. Everything after the click is the person's job: reading the page, closing the cookie banner, finding the size selector, getting through checkout. If the site is awkward, people usually push through.

SEO teams have become very good at the part before the click. Titles, descriptions, sitemaps, structured data, internal links, page speed, crawl budgets. All of that still matters.

What changes when an agent does the clicking

When someone asks an AI assistant to compare three running shoes, check which one is in stock in a size 10, and buy it, there is no results page and often no click by a person. The assistant fetches pages itself. A browser agent may open your site, search your catalog and try to add to cart.

That shifts the work from the person to the agent, and agents don't push through. If the price isn't readable, the agent can't quote it. If a pop-up has no labelled close button, the journey ends there. If bot protection challenges the assistant's request, the person gets a competitor's answer instead of yours. Nobody tells you. The visit simply doesn't turn into anything.

So the question moves from "do we rank?" to "when an agent arrives, can it get in, understand the page, find its way and finish the job?" That's what we mean by agent readiness.

Where SEO already gets you most of the way

The good news for SEO teams is that a lot of the groundwork is shared. Several AgentScore checks will look familiar.

If your SEO is in good shape, you have a head start. But it's a head start, not the finish.

Where SEO stops and agent readiness carries on

Being let in, not just crawled

SEO teams check that Googlebot can crawl. They rarely check how bot protection treats ChatGPT-User, Perplexity-User or a browser agent. A site can rank well and still challenge every AI assistant at the door, because the rules that protect it from scrapers were tuned for a world where the only good bots were search engines. See bot protection and CAPTCHAs.

Content that exists without JavaScript

Google renders JavaScript, so a page built entirely in the browser can still rank. Most AI crawlers and many assistants don't run JavaScript. They read the HTML your server sends, and if the price or description only appears after scripts load, they see an empty page. See why AI agents can't see JavaScript-only content and prices AI agents can read.

A page that can be used, not just read

SEO ends at the landing page. Agent readiness carries on through the page: buttons and links with names, form fields with labels, pop-ups that can be closed, and an add-to-cart button an agent can find and press. These look like accessibility concerns because they largely are. See accessibility work is agent readiness work.

A task that can be finished

The biggest difference is the last step. An agent sent to buy something either completes the purchase or it doesn't. No ranking report tells you which. The only way to know is to send a real agent through the journey and watch. That's what Ghost Agents do, and it's why Task completion carries the most weight in AgentScore.

Measuring what agents bring

SEO is measured in rankings, impressions and organic sessions. Agents mostly don't run your analytics tag, so they don't show up in those numbers. Agent visits live in server and CDN logs, and the visits and sales that AI assistants refer need their own tracking. See how to measure agent traffic and tracking visits and sales that come from AI assistants.

Found versus chosen

Here's the simplest way to put the difference to a leadership team.

• SEO gets you found. It decides whether you're in the answer, and how prominently.
• Agent readiness gets you chosen. It decides whether an agent that found you can confirm the price, check the returns policy and complete the order, or whether it gives up and picks a site it can use.

An assistant that can't read your stock level has no reason to recommend you over a competitor whose stock level is right there in the HTML. Being found and then failing the agent is close to not being found at all.

What this means for SEO teams

This isn't a new discipline that replaces SEO. It's SEO's natural next step, and SEO teams are well placed to lead it. They already own robots.txt, titles, sitemaps and much of the content. They already work with developers on structured data and rendering. What's new is a set of partners SEO hasn't always needed: whoever runs bot protection and the CDN, and whoever owns checkout. We cover that split in who owns agent readiness?

If you want a structured path from there, our 90-day agent readiness plan breaks the work into three phases.

Four categories, 100 points

Every check belongs to one of four categories. Each category has a fixed number of points, set by how much it matters to an AI agent trying to do something for a customer.

The background to these categories is in our guide to agent readiness. This post is about the mechanics.

How a check earns points

Each check ends in one of four results:

• Pass earns the check's full weight.
• Warning earns half. Something works, but agents will stumble on it.
• Fail earns nothing.
• Not tested is left out of the sum entirely. It never counts against you.

Within a category, checks carry different weights, because some problems stop an agent cold and others only slow it down. A category's score is its points multiplied by the share of weight you earned among the checks that could be tested. If Readability's tested checks are worth 17 in total and you earn 13.5 of them, you get 25 × 13.5 ÷ 17, or 19.9 out of 25.

The overall score adds up the categories that could be tested and scales the result to 100. So if Task completion wasn't tested and you scored 20 for Access, 17 for Readability and 11 for Navigability, your AgentScore is 48 out of a possible 70, which is 69.

Most checks are rule-based rather than judged by an AI model, so the same site gets the same score from one scan to the next. When it moves, something on the site changed.

What the bands mean

The one-line verdict at the top of a report also names your biggest issue: the failed check with the highest weight.

The checks, by category

Here is what each category looks at today. The weight is how much the check counts inside its category.

Access

• Bot protection lets AI agents through (weight 4). We request your home page as five AI agents and as a normal browser, and compare. If any agent is blocked or challenged while the browser gets through, it fails. If an agent gets less than half the browser's text, it's a warning.
• robots.txt lets AI assistants and search agents in (3). Blocking one or two assistants or AI search agents is a warning; three or more is a fail. Blocked training crawlers are reported but not scored.
• AI agents can open your product, pricing and cart pages (3).
• AI agents aren't blocked or shown a CAPTCHA at the cart and checkout (3).
• No CAPTCHA or challenge on arrival (2).
• Shoppers can check out without an account (2).
• llms.txt guide for AI (1), Agents can use your site through MCP or WebMCP (1) and Agents can check out through an agentic commerce protocol (1). These are newer standards, so a missing one is a warning, never a fail.

Readability

• Content loads without JavaScript (4): how much of the page's text is in the HTML before scripts run.
• Structured data describes your business and products (3) and Product pages give price and stock in a form agents can read (3).
• Clear page title, description, and headings (2), Valid sitemap (2) and Prices are in the page HTML (2).
• Images have text descriptions (1).

Navigability

• Buttons and links have names (4) and Pop-ups and banners can be dismissed by agents (4).
• Agents can find and press your add-to-cart or sign-up button (3) and Cart and checkout fields are labelled for agents (3).
• Form fields are labelled (2), Page has main and navigation landmarks (2), Clickable things are real buttons and links (2) and Key pages are linked from the home page (2).

Weights can change between scoring versions as we learn what trips agents up. Every report records the scoring version it used, so you can compare like with like.

What the scanner visits

AgentScore reads public pages only. It never adds anything to a cart, submits a form, signs up or places an order. In one scan it looks at:

1. Your home page, three ways: as a browser sees it before JavaScript runs, rendered in a real browser, and as five AI agents (ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot and Googlebot).
2. Your robots.txt, sitemap and llms.txt, plus the well-known addresses where sites describe an MCP server or an agentic commerce profile.
3. Up to three key pages: a product page, a pricing page and the cart. It looks for them the way an agent would, in the links on your home page first, then in your sitemap, then at common addresses such as /pricing and /cart. Each is fetched as a browser and as an AI assistant, and product and pricing pages are rendered in a browser too.
4. Your checkout, found from the link on the cart page or at /checkout, fetched as a browser and as three AI assistants.

Requests under the scanner's own name are signed so you can verify them. The requests that imitate AI agents are not, on purpose. We explain why in Why our scanner sometimes wears other agents' names. How it identifies itself and how to opt out are on the AgentScore bot page.

What isn't scored yet

How to use your score

1. Read the verdict and the biggest issue first. It's the highest-weight failure.
2. Fix failures before warnings, and Access before everything else: if agents can't get in, nothing else matters.
3. Hand each finding to the right person. Most Access problems belong to whoever runs your CDN or bot protection; most Navigability problems belong to developers.
4. Rescan after each fix. In the Ghost Agent Labs app you can rescan whenever you like. The free AgentScore page reuses a domain's result for 24 hours, so repeated requests don't hit your site again.

Then go beyond the score: set up tests of your key journeys with AI agents, so you know when a release breaks them.

Why test with a real agent

Checks for labelled buttons, structured data and bot protection catch the common problems. But journeys fail in ways no rule predicts: a size picker the agent can't operate, a discount pop-up that appears on the second page, a checkout button that only shows after scrolling. People get past these without noticing. Agents often don't.

This is what our team has done for human visitors for years with Ghost Inspector: prove the journeys that make you money still work, and find out when a release breaks one. Ghost Agents bring the same idea to AI agents.

How a test works

You write a goal in plain English, the way a customer might ask an AI assistant. To make it quick, there are ready-made templates for the journeys that matter most:

On each run, the agent opens your site in a fresh browser and repeats a simple loop. It looks at the page as an agent sees it: the address, the visible text, and the buttons, links and fields with their names. It picks one action, such as click, type, choose an option, scroll, go back or finish. Then it acts, and looks again.

How you know it passed

An agent saying "done" isn't proof. So you choose pass conditions that are checked against the final page the agent reaches, without asking an AI model:

• The page address contains some text, such as /checkout.
• The page shows some text, such as "Thanks for signing up".
• The agent reached the payment step. This is the one to use for checkout tests.
• The agent reports the goal is done. This is used only if you set no other condition.

AI agents don't behave the same way every time, so each check runs the journey several times (three is a good default) and reports a pass rate. A test where every run passes is passing. One where none pass is failing. One where results disagree is flaky: agents can do the journey, but not reliably, which for a customer's assistant can mean the same as not at all.

Tests run when you click "Run now", every day, or every hour. When one fails, you can be alerted by email or Slack.

Replays: see exactly where it got stuck

Every run is recorded. The replay opens on the step where things went wrong, with a screenshot of what the agent saw, the action it took, and its reasoning, which often says in plain words what it couldn't find or press. If a journey used to work, you can compare the failed run with the last passing one to see what changed.

That turns "agents can't check out" into something a developer can act on: "the cookie banner covers the checkout button and has no labelled close button".

Safe by design

A test agent that clicks around a live store has to be careful. Ghost Agents follow hard rules that the model can't talk its way past:

• They stop as soon as a payment form appears, and never enter card details. Reaching that point is how a checkout test passes.
• They refuse buttons that place orders, pay, or delete or cancel an account.
• They type only the test data you give the test, such as a test email address.
• They stay on the site being tested.
• Each run stops after 40 steps or five minutes, whichever comes first.

They also identify themselves. Every request carries GhostAgent/1.0 (+https://ghostagentlab.com/ghost-agent) at the end of a normal Chrome user agent and is signed with Web Bot Auth, so your CDN can verify it's really us. We explain signing in Why we sign every request our agents send, and the Ghost Agent page covers how to allow or opt out.

What Ghost Agents don't do yet

We'd rather you knew the limits up front:

• They don't feed AgentScore yet. Task completion is the AgentScore category these tests will fill. Until they're connected, the score covers Access, Readability and Navigability. See How AgentScore works.
• One agent per run. Every agent gets the same instructions and tools, so results from different AI models can be compared, but running several side by side in one check isn't available yet.
• No release triggers yet. Tests run on demand or on a schedule, not automatically after each deploy.
• No payments. By design, a checkout test proves an agent can reach payment. It never completes one.

Where to start

For help choosing journeys and writing good goals, read How to test your key journeys with AI agents.

How a browser agent sees a page

A person looks at a page and sees layout, color and images. A browser agent, the kind that opens your site in a real browser to search, fill in forms and add to cart, needs something it can reason about and act on. It has two main options.

• Screenshots. The agent looks at an image of the page and decides where to click. This works, but it's slow, and it struggles with small icons, overlapping elements and anything that looks clickable but isn't.
• The accessibility tree. Browsers already build a structured version of every page for assistive technology: each heading, link, button and form field, with its role (what kind of thing it is), its name (what it's called) and its state (checked, expanded, disabled). Many agents read this tree, often alongside screenshots, because it tells them exactly what can be clicked and what each thing is for.

The tooling used to build agents reflects this. Playwright's MCP server, a widely used way to give an AI model control of a browser, hands the model an accessibility snapshot of the page rather than pixels. In that snapshot, a button is a line like button "Add to cart". A button with no name is just button, and the agent has to guess.

<!-- What the agent can use -->
<button type="submit">Add to cart</button>
<button aria-label="Open cart"><svg aria-hidden="true">…</svg></button>
<!-- What it can't -->
<div class="btn" onclick="addToCart()"><svg>…</svg></div>

The last example has no role and no name. A screen reader user can't use it, and an agent reading the accessibility tree may not know it exists.

Where WCAG and agent readiness overlap

The Web Content Accessibility Guidelines (WCAG) 2.2 are the standard most accessibility programs work to. Several of their success criteria line up closely with AgentScore checks.

The match isn't exact. WCAG asks more of each criterion than our checks test for, and our checks look at things from an agent's point of view. But the direction is the same: give every control a role and a name, label every field, describe every meaningful image, and structure the page so it can be navigated without seeing it.

Our guide to buttons, links and forms agents can use covers the fixes in detail, and alt text for AI agents covers product images.

Common accessibility gaps that also stop agents

• Icon-only buttons with no label. The cart, search and menu icons in a header are the usual culprits. Add visible text or an aria-label.
• Placeholder text used as a label. It disappears when the field is filled in and isn't a reliable name. Use a real <label>.
• Clickable boxes. A div with a click handler looks like a button to a person and like plain text to the accessibility tree. Use <button> or <a href>.
• Menus that only open on hover. Neither keyboard users nor many agents can hover. Open them on click and focus too.
• Custom size and color pickers. If a swatch has no name and no selected state, an agent can't tell which size it chose. Native radio buttons or properly labelled custom controls fix this.
• Pop-ups without a labelled close button. See cookie banners and pop-ups that trap AI agents.

Where they differ

Accessibility work won't cover everything, and some of it doesn't matter to agents at all.

Agent readiness needs more than accessibility. A perfectly accessible site can still block AI assistants in robots.txt, challenge them with bot protection, or load its prices with JavaScript that many AI crawlers never run. Screen readers work inside a real browser, so they see JavaScript content; many agents don't. Structured data, sitemaps and llms.txt matter to agents and have little to do with accessibility. That's why AgentScore has Access and Readability categories as well as Navigability.

Some accessibility work matters less to agents. Color contrast, text resizing, captions and motion settings are essential for people and largely irrelevant to software. Don't use agent readiness as a reason to deprioritize them. They serve customers, and in many places they're a legal requirement.

Bad ARIA hurts both. The W3C's own guidance is that no ARIA is better than bad ARIA. An aria-label that says "button", a role="button" on something that can't be pressed, or aria-hidden="true" on a real control misleads screen readers and agents alike. Native HTML elements are almost always the better choice.

How to use this

1. Ask your accessibility lead for the latest audit. Unlabelled controls, missing form labels and keyboard traps in that report are agent readiness issues too. Fixing them pays twice.
2. Add an agent's view to your testing. Run AgentScore and compare its Navigability findings with your audit. Then send a Ghost Agent through checkout to see whether a real agent can finish.
3. Make it part of the definition of done. New components ship with names, labels and native elements. That's cheaper than fixing them later, for people and agents alike.

Accessibility and agent readiness come from the same idea: a website should work for visitors who don't see it the way its designers do. More and more of those visitors are software acting for a person.

The question we're trying to answer

One of the most common reasons an AI agent fails on a website has nothing to do with the website's design. The agent is turned away at the door. A bot protection rule, a firewall setting or a CDN feature decides that anything calling itself an AI agent gets a challenge page, an error, or a stripped-down page with no prices.

Nobody notices, because people never see it. The site works perfectly in a browser. The only way to find out is to knock as the agent and compare what comes back with what a browser gets.

We can't send the real ChatGPT to your site on demand. So for a handful of requests, our scanner sends the same User-Agent header those agents send, and compares the answer with a normal browser's.

Exactly what we send

The imitated requests use the user agent strings those operators publish, so they look the way the real thing does. We picked these agents because they're the ones that fetch pages for people in real time or power AI search, which are the visits most likely to turn into customers.

Training crawlers such as GPTBot and ClaudeBot are handled differently. We read your robots.txt rules for them, along with every other AI agent we track, but we don't request pages as them. Blocking training is a business choice, and it doesn't cost points. Our guide to blocking or allowing AI crawlers covers that decision.

How we compare the answers

For each imitated request, we line the response up against the browser's:

• Blocked: the agent got an error (HTTP 400 or above) where the browser didn't, or a challenge page such as "Just a moment" or "Verify you are human" where the browser got the real page.
• Degraded: the agent got less than half the visible text the browser got, which usually means a placeholder or stripped-down page.
• Same page: neither of the above.

On the home page, any blocked agent fails the Bot protection lets AI agents through check, and a degraded one is a warning. A similar comparison, looking for errors and challenge pages (and, at checkout, CAPTCHAs), runs on your product, pricing and cart pages for AI agents can open your product, pricing and cart pages, and on the cart and checkout for AI agents aren't blocked or shown a CAPTCHA at the cart and checkout. How AgentScore works explains how those checks add up.

Why those requests aren't signed

Everything the scanner sends under its own name is signed with Web Bot Auth, so your CDN can prove it came from us. We explain how in Why we sign every request our agents send.

The imitated requests are deliberately unsigned. If we signed them, a CDN that recognizes and trusts our signature might wave them through, and we'd be measuring how your site treats Ghost Agent Labs, not how it treats ChatGPT. Unsigned, they get exactly the treatment any request with that name gets.

What the result can't tell you

Our imitation is honest about one thing it can't do: it can't pass identity checks. The requests come from our servers, not from OpenAI's, Anthropic's, Perplexity's or Google's, and they carry none of those companies' signatures.

So if your bot protection verifies agents properly, checking published IP ranges or reverse DNS and blocking impostors, it may block our imitation too. That's the right behavior, and it will show up as a failed check. When that happens:

1. Look at the evidence in the report. It shows which agents were blocked and what status they got.
2. Check your CDN or bot protection settings: is the rule "block requests that claim to be an AI agent but fail verification", or "block AI agents"? The first is good practice. The second turns customers away.
3. Confirm in your logs, or on the verification page in the Ghost Agent Labs app, that verified requests from the real agent are getting through.

If the real agents get through and only impostors are blocked, you're in good shape, whatever the check says. Our guides to telling whether an AI crawler is real and bot protection that lets AI agents through cover how to set that up.

The opposite limit applies too. A site that treats these agents well when they come from our servers will probably treat the real ones well, but real products differ in details we can't copy, such as where their requests come from. Treat the result as a strong signal, not a guarantee.

Keeping it small

Imitated requests are a small part of a scan: a few page loads per agent at most, never a crawl. The scanner reads public pages only and never adds to a cart, submits a form or places an order. On the free AgentScore page, a domain's result is reused for 24 hours, so repeated requests for the same site don't cause repeated visits.

Start by splitting the question

The major AI companies each run several agents, and each has its own name so you can treat them differently.

Our robots.txt guide has the longer list and the exact names. The point for decision-makers is simpler: you can say yes to some and no to others.

The case for each, in business terms

Assistant fetchers: almost always allow

Each visit stands in for a real person asking about you right now: "Is this in stock?", "What's their returns policy?", "Which of these three is cheapest?". Block them and the assistant answers without you, or recommends a competitor. For a store or a software company, this is the closest thing to a customer walking in.

AI search crawlers: allow, unless you also opt out of search

These build the index AI search draws on. They're the AI equivalent of being in Google's index, and they're how you get cited and linked. Few businesses that want search traffic would block Googlebot. The same logic applies here.

Training crawlers: a real choice

This is the one worth debating. Allowing training means your content may shape what AI models know, including about your brand and products. Blocking it keeps your content out of future training sets, and leaves AI search and assistants unaffected, because they use different names.

• Lean towards blocking if your content is the product: publishers, research, courses, original reviews, anything you license or sell.
• Lean towards allowing if your content mainly exists to sell something else: product pages, help centers, pricing. Being well understood by AI models is usually worth more to you than the content itself.

Either answer is legitimate. That's why AgentScore reports blocked training crawlers but doesn't take points off for them.

Browser agents: you can't block them by name, so make them work

Browser agents mostly look like a normal browser, so robots.txt rules by name don't reach them. The useful question isn't whether to block them but whether they can complete a purchase or sign-up when they arrive. That's what Ghost Agent tests check.

A quick decision guide

Three mistakes that cost more than the decision

1. Copying a "block all AI" list. These lists usually include assistant fetchers and AI search crawlers, so you drop out of AI answers along with training.
2. Deciding in robots.txt and forgetting the CDN. Bot protection, firewall rules and one-click "block AI bots" settings can override a friendly robots.txt. The agent never gets far enough to read it. See bot protection that lets AI agents through.
3. Trusting the name. Anyone can claim to be ChatGPT or Googlebot. robots.txt is a request, not a lock: well-behaved agents follow it and scrapers ignore it. Stopping abusive traffic, including impostors using trusted names, has to happen at your CDN, by verifying who's really asking. See how to tell if an AI crawler is real.

One more nuance: some operators treat an assistant fetch, made because a person asked about a specific page, like a person clicking a link, and say robots.txt may not apply to it. If you truly need to stop those visits, that's a CDN rule, not a robots.txt line. Most businesses want those visits most of all.

What to do this month

The default for most businesses is simple: let the agents that bring customers in, make a deliberate choice about training, and enforce both where it actually counts.

Before you start

Name one person to own the plan. It doesn't have to be a developer; on most teams it's someone in e-commerce, digital or SEO who can get time from the people who make the changes. Our post on who owns agent readiness shows who usually fixes what.

Then agree what "done" means. We suggest three outcomes by day 90:

• No critical AgentScore findings on your home page and key product pages.
• At least one money journey, such as product to checkout, tested by a real AI agent on a schedule.
• A monthly view of agent traffic and AI referrals that leadership can read.

Notice that none of these is a target score. Scores are useful for tracking progress, but a site that loses ten points to an optional new standard can still serve agents well, and a site that scores well can still fail at the size selector. Aim at the outcomes.

Days 1–30: let agents in and get a baseline

The first month is about Access, because nothing else matters if agents are turned away at the door. It's also when you set the baseline you'll measure against.

Week 1: measure

1. Run AgentScore on your home page and save the report. How AgentScore works explains what's behind each category.
2. Connect a data source, such as your CDN or server logs, so you can see which AI agents visit and how your site answers them. See how to measure agent traffic.
3. List your three most valuable journeys: for most stores, search to product, product to cart, and cart to checkout.

Weeks 2–4: fix Access

Make one policy decision this month too: what you'll do about AI training crawlers, as distinct from the assistants that fetch pages for shoppers. Our decision guide walks through it. Settle it early, because it affects robots.txt and bot protection rules at the same time.

Days 31–60: make pages readable and usable

Month two covers Readability and Navigability: whether agents can understand your pages, and whether they can find their way around them. Most of this is developer work, so get it into the sprint plan early in the month.

Readability

• Content loads without JavaScript and Prices are in the page HTML. The highest-value fixes for most stores. See JavaScript-only content and prices AI agents can read.
• Structured data and product pages give price and stock in a form agents can read. See product data AI shopping agents can read.
• Clear page title, description, and headings, image text descriptions and a valid sitemap. Usually SEO and content work. See titles, descriptions and headings, alt text for AI agents and XML sitemaps.
• Policy pages for shipping and returns that an assistant can quote accurately. See shipping, returns and FAQ pages.

Navigability

• Buttons and links have names, form fields are labelled and clickable things are real buttons and links. See buttons, links and forms agents can use. If you have an accessibility audit, start there: much of it overlaps.
• Pop-ups and banners can be dismissed by agents. See cookie banners and pop-ups.
• Main and navigation landmarks and key pages linked from the home page. See link your key pages from the home page.
• Site search and filters that agents can operate. See site search and filters AI agents can use.

Days 61–90: prove agents can finish the job

Month three is about Task completion, the category that carries the most weight. Checks can tell you a page looks usable. Only a real agent can tell you whether it is.

1. Test your money journeys. Set up Ghost Agent tests for the journeys you listed in week one, starting with product to checkout. Each check runs the agent several times, because agents vary, and Ghost Agents stop before paying. See how to test your key journeys with AI agents.
2. Fix where they fail. The usual sticking points are the add-to-cart button, option selectors, checkout fields and a challenge at the cart. The relevant checks are Agents can find and press your add-to-cart or sign-up button, Cart and checkout fields are labelled for agents, and AI agents aren't blocked or shown a CAPTCHA at the cart and checkout. See agent-ready checkout.
3. Review guest checkout. An agent buying for someone can't easily create an account for them. See guest checkout: why AI shopping agents need it.
4. Look ahead. Ask your platform and developers where you stand on MCP and WebMCP and on agentic commerce protocols. These are early and still changing, so they're weighted lightly in AgentScore. A decision and a plan are enough for now.
5. Set up reporting. Track the visits and sales AI assistants send you, alongside agent traffic from your logs. See tracking visits and sales that come from AI assistants.

After day 90

Agent readiness drifts. A new pop-up campaign, a bot protection rule change or a redesigned product page can undo months of work in one release. Keep three habits:

• Rescan after every significant release, and at least monthly.
• Keep Ghost Agent tests running on your money journeys, and add one when you launch a new one.
• Add agent checks to your definition of done: named controls, labelled fields, prices in the HTML, and no new challenge pages on key paths.

Start with a scan. It takes under a minute, and it'll tell you which month of this plan needs the most attention.

Why it falls between teams

Agent readiness touches four layers of a website, and each layer has its own team.

• The door: robots.txt, bot protection, CDN rules and rate limits decide whether an agent gets in.
• The content: titles, descriptions, product data and policies decide whether it understands what it finds.
• The page: buttons, forms, pop-ups and rendering decide whether it can move around and act.
• The transaction: cart, checkout and accounts decide whether it can finish the job.

No one team sees all four. Each sees its own layer working, and the failure shows up somewhere else: a lost recommendation, or an order that never happened.

Who usually fixes what

In the Ghost Agent Labs app, each AgentScore finding names a suggested owner under “Usually fixed by”, and the CSV export includes it too. Grouped together, the defaults look like this.

Task completion, which comes from real agents running journeys, usually lands with developers once a test shows where an agent gets stuck. But the decision about which journeys matter belongs to the business.

These are defaults, not rules. On a hosted platform, the "developer" fixes for a product page may really be theme settings an e-commerce manager can change. At a small company, one person may hold three of these roles.

The roles in more detail

E-commerce and digital leaders

They own the outcome: whether AI agents can find, choose and buy from the site. That makes them the natural overall owner. They decide which journeys matter, set priorities when fixes compete for developer time, and own the platform settings that shape checkout, such as guest checkout. See guest checkout: why AI shopping agents need it.

Marketing and SEO

They own most of what agents read before they act: robots.txt, titles and headings, alt text, sitemaps and llms.txt. They also own measurement, because AI referrals belong next to search and social in channel reporting. See SEO got you found. Agent readiness gets you chosen and tracking visits and sales that come from AI assistants.

Developers

They own the largest number of fixes: rendering content and prices in the HTML, structured data, named controls, labelled fields, dismissible pop-ups and checkout fields. Much of this overlaps with accessibility, so the same people and practices often apply. See accessibility work is agent readiness work. Developers also run agent tests as part of release checks, and evaluate newer interfaces such as MCP and WebMCP.

Security and CDN

They own the door. Bot protection, WAF rules, challenge pages and rate limits are where agents are most often blocked by accident, because those rules were written to stop scrapers. This team's job isn't to let everything in. It's to tell real agents from impostors, by verified identity rather than by name alone, and to treat the agents the business wants accordingly. See bot protection and CAPTCHAs, how to tell if an AI crawler is real and rate limits for AI agents.

Legal

Legal doesn't fix findings, but it shapes several decisions the other teams can't make alone:

• Whether to allow AI training crawlers, which is a content-licensing question as much as a technical one. See should you block AI crawlers?
• What your terms of service say about automated access, and whether they accidentally forbid the agents you want.
• The terms and liability around purchases made by an agent for a customer, before you adopt an agentic commerce protocol.
• Accessibility obligations, which often share fixes with agent readiness.

Making it work in practice

1. Name one accountable owner. Usually the head of e-commerce or digital. They don't make every fix; they make sure each one has someone.
2. Route findings by owner. Export the findings as CSV and send each group to its team. Each finding explains why it matters in plain language, which helps when the team receiving it doesn't think of AI agents as its problem.
3. Agree the policy decisions once. Training crawlers, verified agents through bot protection, and guest checkout are decisions, not tickets. Settle them in one meeting with SEO, security, e-commerce and legal in the room.
4. Make failures visible to the owner. Run Ghost Agent tests on your money journeys and send alerts to the person who can act, not a shared inbox.
5. Review monthly. AgentScore by category, Ghost Agent pass rates, agent traffic and AI referrals, on one page.

For a phased version of this, see our 90-day agent readiness plan. Or start by running AgentScore and seeing whose name comes up most.

A new kind of visitor

When someone asks an AI assistant "which of these running shoes is best for flat feet, and is it in stock in a 10?", a person never visits your site. Software does. That software comes in three broad kinds:

• Training crawlers, such as GPTBot and ClaudeBot, collect pages to teach AI models. They shape what AI knows about you over months.
• AI search and assistant fetchers, such as OAI-SearchBot, ChatGPT-User and Perplexity-User, fetch your pages in the moment to answer a question. They decide whether you're the answer today.
• Browser agents drive a real browser for a person: searching your catalog, filling in forms, adding to cart, and in some cases checking out.

The last two are the ones that matter most to revenue, because each visit stands in for a real customer with a real intent. If the agent can't find the price, read the return policy, or get past a pop-up, the customer doesn't get a worse experience on your site. They get a recommendation for someone else's.

Why nobody noticed

Most websites have no idea how many agents visit, for a simple reason: analytics tools count visitors with a JavaScript tag, and most agents never run it. Crawlers and assistant fetchers request the HTML and leave. Your dashboards show a quiet day while your server logs show a busy one.

The agents that do run JavaScript often look like ordinary browsers, so they blend into the human numbers. Either way, the question "did the agent get what it came for?" goes unanswered.

What goes wrong

When we test sites with real agents, the failures fall into the same few patterns:

1. They're turned away at the door. A robots.txt rule written years ago, or bot protection that challenges anything that isn't a person, blocks the assistants that were trying to send customers.
2. They can't read the page. Prices and product details only appear after JavaScript runs, and there's no structured data to fall back on.
3. They can't find their way. Unlabelled icon buttons, menus that only open on hover, and cookie banners that cover the page stop an agent the way a locked door would.
4. They can't finish the job. The agent finds the product but can't choose a size, or reaches the cart but can't find the checkout button.

None of these show up in a normal QA pass, because a person gets through every one of them without noticing.

The gap in today's tools

Two kinds of tools touch this problem, and neither solves it.

Bot management decides who to block. It's essential, and it's built to keep bad traffic out, which means good agents are often collateral damage. AI visibility tools track what chatbots say about your brand. That's useful too, but it measures the answer, not whether an agent could use your site to get there.

What's missing is the thing we've always done for human visitors: test the experience. Can an agent actually complete the journeys that make you money, and does it still work after this week's release?

Agent experience

We think agent experience will become a discipline of its own, next to user experience and SEO. It has three parts, and they're how we've built Ghost Agent Labs:

• Observe. See every crawler, assistant and browser agent that visits, from server and edge data rather than a JavaScript tag, and check which ones are real.
• Test. Send real AI agents through checkout, sign-up and search on a schedule, and get alerted when they fail where a person would succeed.
• Improve. Score agent readiness, fix the highest-impact problems first, and re-test to prove the fix worked.

It's the same idea behind Ghost Inspector, which has tested websites for human visitors for years, applied to the visitors that aren't human.

What to do this week

Your next million visitors won't be human. The sites that are ready for them will win the customers they bring.

The problem with user agents

Every request a browser or bot makes carries a User-Agent header. Ours look like this:

AgentScore/1.0 (+https://ghostagentlab.com/agentscore/bot)
GhostAgent/1.0 (+https://ghostagentlab.com/ghost-agent)

That's useful, and it's a courtesy we'll keep. But it proves nothing. A scraper can send GPTBot, ClaudeBot or AgentScore/1.0 just as easily as the real thing, and plenty do. So site owners face a bad choice: trust the label and let impostors in, or block the label and turn away the agents they actually want.

The traditional fix is to check where the request came from: published IP ranges, or a reverse DNS lookup that resolves back to the operator's domain. That works for big crawlers with fixed infrastructure. It works badly for browser agents running in the cloud, where IP addresses change all the time, and it puts the burden on every website to keep lists up to date.

What Web Bot Auth does instead

Web Bot Auth is a proposal, now being worked on at the IETF and supported by Cloudflare's verified bots program, that lets an agent prove who it is the same way websites prove who they are: with a public key.

1. The agent operator publishes its public keys in a directory at a well-known address on its own domain. Ours is https://app.ghostagentlab.com/.well-known/http-message-signatures-directory.
2. Every request carries three extra headers: Signature-Agent (where to find the keys), Signature-Input (what was signed, when, and with which key) and Signature (the signature itself), following HTTP Message Signatures, RFC 9421.
3. The website or its CDN fetches the key once, checks the signature, and knows for certain the request came from the operator, whatever IP address it arrived from.

Anyone can copy a user agent. Only the holder of the private key can produce the signature.

How we do it

We sign with an Ed25519 key. Each signature covers the site's host name and our Signature-Agent header, carries a fresh random nonce, and expires after 60 seconds, so a captured signature can't be replayed against another site or reused later. The key ID is the key's standard JWK thumbprint, and the key directory itself is signed too, so nobody can swap in their own keys.

That last row matters. Part of what AgentScore measures is whether your bot protection treats AI agents differently from people. If we signed those requests, a CDN that trusts us would wave them through and the test would tell you nothing.

Why this matters beyond us

Agents are becoming customers' representatives. When someone asks an assistant to reorder printer ink or book a table, the agent that arrives at your site is acting for a real person with a real wallet. The sites that win are the ones that can say yes to those agents confidently, without opening the door to every scraper wearing a borrowed name.

Signed requests make that possible. A site can allow verified agents through a challenge page, give them a lighter rate limit, or simply count them accurately in analytics. None of that works if the identity is a string anyone can type.

Checking it's really us

The details of what our agents do, what they won't do, and how to opt out are on the AgentScore bot and Ghost Agent pages. We don't use fixed IP addresses yet, so if you want to allow us, please do it by signature or user agent rather than by IP.