AgentScore bot

About the AgentScore scanner

AgentScore is a free test that checks how well AI agents can use a website. When someone requests a score for a site, our scanner visits a small number of that site's public pages.

How it identifies itself

Requests for robots.txt, sitemaps, llms.txt and well-known files, and checks for common pages such as /pricing and /cart (with one made-up address such as /agentscore-3f9a1c2b7d4e, to see what your site sends for a page that doesn't exist), use this user agent:

AgentScore/1.0 (+https://ghostagentlab.com/agentscore/bot)

To see your site the way its visitors do, we also request a few pages under other user agents and compare the responses:

  • As a desktop browser: your home page and llms.txt, and up to one product, pricing and cart page and your checkout page. We also open the home page, product, pricing and checkout pages in a headless browser, which loads their scripts and images, to see them after JavaScript runs.
  • As AI assistants and search agents: your home page once each as ChatGPT-User, Claude-User, Perplexity-User, OAI-SearchBot and Googlebot; your product, pricing and cart pages as ChatGPT-User; and your cart and checkout pages as ChatGPT-User, Claude-User and Perplexity-User.

These requests aren't signed, so your site treats them exactly as it would those visitors. The ones under AI agents' names carry a gal_scan parameter at the end of the address, with a code signed for your domain and the minute it was sent, so you can tell them from the real agents in your logs. Ghost Agent Lab leaves them out of your traffic, so they never show as impostors. We never request pages as training crawlers such as GPTBot or ClaudeBot; we only read the rules your robots.txt sets for them.

How to check it's really us

Requests made under the AgentScore user agent are signed with Web Bot Auth (HTTP Message Signatures). Each carries a Signature-Agent header pointing to https://app.ghostagentlab.com, where our public key is published at /.well-known/http-message-signatures-directory. Anyone can copy a user agent, but only we can produce the signature, so services that check it, such as Cloudflare's verified bots, can tell our traffic from impostors.

We don't use fixed IP addresses yet, so please allow us by signature or user agent rather than by IP.

What it does

  • Visits at most about 20 pages per scan.
  • Reads your robots.txt before anything else, and stops without requesting any other page if it disallows AgentScore.
  • Presses your add-to-cart button once, when your cart and checkout show no form while the cart is empty, then looks at the cart and checkout. If a cookie banner or pop-up covers that button, it presses the banner's close or accept button first, as a shopper would. That item stays in a cart only our scanner's browser session can see, and is gone when the scan ends.
  • Never submits payments, orders, account sign-ups, or messages, and never types real personal data.
  • Reuses results for 24 hours, so repeated requests for the same site don't cause repeated visits.

If your site blocks the scan

If your bot protection turns AgentScore away even when it visits your home page as an ordinary browser (HTTP 401 or 403), the scan stops there and says so, rather than scoring the block page. That usually means a rule blocks traffic from cloud servers. AI assistants and shopping agents browse from cloud servers too, so they may be turned away the same way. Look for rules that block cloud or data-center traffic, and allow verified bots. If the site answers that we're making too many requests (HTTP 429), wait a few minutes and scan again.

How to opt out

Add this to your robots.txt to stop AgentScore scanning your site:

User-agent: AgentScore
Disallow: /

The scan then stops after reading robots.txt, and whoever requested it is told your site has asked not to be scanned. It has to be a rule for AgentScore by name: a User-agent: * rule on its own doesn't opt you out, because AgentScore's job is to show how your site treats agents.

To have your domain blocked from AgentScore without changing robots.txt, email bot@ghostagentlab.com from an address at that domain.