Why AgentScore scores with rules, not an AI model

AgentScore measures how well AI agents can use your website, yet no AI model decides your score. That's deliberate. Here's why we score with rules, what rules get wrong, and where we do use real AI agents.

What "rule-based" means here

Every AgentScore check is ordinary code with a fixed test and a fixed threshold. It fetches a page, reads the HTML or the rendered page, and compares what it finds against a rule. A few examples:

• Content loads without JavaScript passes when at least 70% of the rendered page's text was already in the HTML, warns from 30% to 70%, and fails below that.
• Bot protection lets AI agents through requests your home page as four AI agents, as a browser and as Googlebot, and fails when an agent is blocked or challenged while the browser and Googlebot get through.
• Cart and checkout fields are labelled for agents counts fields with no label, fields labeled only by placeholder text, and fields missing the right input type or autocomplete value.

Weights are fixed per check, results are pass, warning, fail or not tested, and even the one-line verdict at the top of your report is a template filled in with the score band and your highest-weight failure. We checked the scanner's code for this post: it has no AI model in it and calls none. Its building blocks are an HTML parser, a robots.txt parser, an HTTP client and a headless browser.

That may sound old-fashioned for a product about AI. It's the point.

Same site, same score

A score is only useful if a change in it means something. When your AgentScore moves from 71 to 78, you want to know it's because your developers labeled the checkout fields, not because the scorer had a different day.

Language models don't behave the same way every time. Ask one to judge the same page twice and you can get two answers. And the model behind a product changes when its maker ships a new version, so a score could drift with no change to your site at all.

Scored by rules

• Same pages give the same result
• Changes only when your site changes, or when we publish a new scoring version
• Each report records its scoring version

Scored by a model

• The same page can be judged differently twice
• Can drift when the model is updated
• Hard to say which change moved the score

Rules make three things possible that a team relies on. You can track progress month to month, as agent readiness KPIs suggests. You can compare yourself with competitors scanned the same way. And you can rescan after a fix and know the difference is the fix.

Two honest caveats. Your site isn't perfectly still: a slow page, a pop-up that shows on some visits or a product that just sold out can change a result. And we do change the rules, when we learn something. When we do, we publish a new scoring version, and scores from different versions shouldn't be compared.

Every finding points at something you can change

A rule knows why it failed, so it can tell you. A finding doesn't say "the model thinks your checkout is confusing". It says something like "2 of 9 fields have no label (email, postal code)", names the page, and comes with a fix written for that exact rule.

That turns a score into a work list. Each finding can go to the person who owns it, whether that's whoever runs your CDN or a front-end developer, and it can be checked again when it ships. Reading your AgentScore report walks through it.

A rule knows why it failed, so it can tell you what to change.

Fast and cheap enough to run often

A typical scan takes 5 to 20 seconds and makes about 25 requests beyond your home page. That's what lets us offer AgentScore free, lets you rescan after every fix, and let us run it on 600 websites in a single day for our store study. Sending an AI model through every page of every scan would be slower and far more expensive, and it would make the study's figures impossible to repeat.

What rules get wrong

Rules have a weakness: they only know what we thought of. Running AgentScore on 600 real stores showed how much that matters. An English-only word list missed add-to-cart buttons on stores in other languages, and one rule flagged good buttons as fake: 57% of stores where we found the button were flagged, when the real figure was 24%. What scanning 600 stores taught us covers each bug and its fix.

Later, we found that sites with well-configured bot protection were marked down for turning away our test requests, which use AI assistants' names from our own servers. We fixed that too.

Two things make these mistakes manageable. A wrong rule is wrong the same way every time, so it shows up clearly in aggregate and can be fixed for everyone at once. And when a rule can't tell, it says "not tested" rather than guessing, and a check that isn't tested never costs points.

The bigger limit is that rules can't predict everything that stops a real journey: a size picker an agent can't operate, a discount pop-up on the second page, a checkout step that only appears after the first.

Where AI models do belong

That last limit is where real AI agents earn their place. AgentScore's fourth category, Task completion, is worth 30 points and needs an AI agent to actually try a task, so the free scan scores it only when its agent runs, and a failed task caps the score at 59.

• AgentScore Rules. Can agents get in, read your pages and find their way? Same site, same score.
• Ghost Agents Real AI agents run your journeys, stop at payment, and record every step.
• AI visibility (beta) How Claude answers shoppers' questions about you, which only a model can show.

In the Ghost Agent Lab app, Ghost Agents send real AI agents through journeys such as finding a product and reaching checkout. Because agents vary, each check runs the journey several times and reports a pass rate. Even there, we keep the verdict out of the model's hands where we can: you set pass conditions, such as reaching the payment step, and they're checked against the final page without asking a model.

That's the pattern: use a model to do the thing a customer's agent would do, and use rules to judge and score it. How to test your key journeys with AI agents covers the practical side.