Agent readiness glossary: the terms you'll hear, in plain English

Agent readiness comes with a lot of new words, and some old ones used in new ways. This glossary explains the terms you'll meet in our guides, in AgentScore reports and in conversations with your developers and vendors, in plain English and with what each one means for your business.

The agents

AI agent. Software that uses an AI model to do something on the web for a person or a company: read a page, answer a question, compare products or complete a task. In our guides, “agent” covers everything from crawlers to agents that shop.

Training crawler. A bot that collects pages to train AI models, such as GPTBot or ClaudeBot. Blocking it doesn't stop AI assistants visiting for a person.

AI search crawler. A bot that indexes pages so AI search and assistants can cite and link to them, such as OAI-SearchBot or PerplexityBot.

Assistant fetcher. An AI assistant fetching a page in real time because a person asked it something, such as ChatGPT-User or Claude-User. Each visit stands in for a customer.

Browser agent. An agent that opens your site in a real browser and clicks, types and scrolls like a person, to book, sign up or buy. Also called a computer-use agent or autonomous agent.

User agent. The name a visitor sends with every request to say what it is, such as a browser version or GPTBot. Easy to fake, so it shouldn't be trusted on its own.

For how these differ and how to treat each, see the AI agents that visit your website.

Getting in: access

robots.txt. A file at the root of your site that tells crawlers and agents which pages they may visit. Well-behaved agents follow it; scrapers ignore it. See robots.txt for AI agents.

Bot protection. Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors. Set too strictly, they turn away AI assistants as well as scrapers. See bot protection and CAPTCHAs.

CDN. Content delivery network: the service that sits in front of your site and delivers it quickly around the world. It's often where bot rules live, and where agent traffic is recorded. See setting up your CDN for AI agents.

CAPTCHA or challenge. A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page. Agents can't pass them, so one on arrival or at checkout ends the visit.

Rate limit. A cap on how many requests a visitor can make in a given time. Fair limits stop scrapers without turning away agents. See rate limits for AI agents.

Verification. Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature. See how to tell if an AI crawler is real.

Impostor (spoofed agent). A request that uses a trusted agent's name but fails verification. Usually a scraper trying to get past bot protection.

Forward-confirmed reverse DNS. A verification check that an IP address's hostname belongs to the operator, and that the hostname points back to the same address.

Web Bot Auth. A new standard in which agents sign their requests cryptographically, so sites can prove who sent them. Our own scanner and Ghost Agents sign theirs. See why we sign every request.

llms.txt. A short Markdown file at /llms.txt that tells AI what your business does and links to your key pages. An emerging standard. See how to write an llms.txt file.

Guest checkout. Letting shoppers buy without creating an account. An agent buying for someone can't easily sign up for them. See guest checkout.

Understanding pages: readability

Raw HTML. The page exactly as your server sends it, before any JavaScript runs. It's all that many crawlers and assistant fetchers ever see. To look at it, use your browser's “View page source”, not the inspector.

Server-side rendering (SSR) and static generation. Ways of building pages so the content is already in the raw HTML, rather than added afterwards by JavaScript. See why AI agents can't see JavaScript-only content.

Structured data (schema.org, JSON-LD). Machine-readable facts in your page code, such as a product's name, price and stock, written in the shared schema.org vocabulary, usually as JSON-LD. Agents can read it without guessing. See product data AI shopping agents can read.

XML sitemap. A file listing the pages you want found, so crawlers and agents don't depend on following links. See XML sitemaps for AI agents.

Canonical URL. The one address you name as the main version of a page that can be reached in several ways. It stops agents treating duplicates as different products. See canonical URLs.

Alt text. A short text description of an image. Agents can't see pictures, so this is how they learn what a product photo shows. See alt text for AI agents.

Finding the way: navigability

Accessibility tree. The structured view of a page that screen readers and many browser agents use: its headings, links, buttons and form fields, with their names. If something isn't in it, many agents can't use it. See what an AI agent sees.

Accessible name. The name a button, link or field has in the accessibility tree, from its text, label or aria-label. An icon-only button with no name is invisible to agents.

Landmarks. Labels for the main regions of a page, such as header, navigation, main content and footer, that let agents skip straight to what they need.

Overlay. Anything that covers the page, such as a cookie banner, newsletter pop-up or chat window. If an agent can't close it, the journey ends there. See cookie banners and pop-ups.

WCAG. The Web Content Accessibility Guidelines, the standard for making sites usable by people with disabilities. Much of it helps agents too. See accessibility work is agent readiness work.

Getting it done: tasks and commerce

Journey. A task a visitor sets out to complete on your site, such as finding a product and reaching checkout, or booking a call. The journeys that make you money are the ones to test first.

Task completion. Whether an agent can actually finish a journey. It's the AgentScore category worth the most points, and the only way to measure it is to try. See how to test your key journeys.

MCP (Model Context Protocol). A standard way to give AI agents direct tools for your site, such as search, product details and cart, instead of making them click through pages.

WebMCP. A proposal for offering those same tools from inside your web pages, so a browser agent can call them while it's on your site. See MCP and WebMCP.

Agentic commerce. Purchases that an AI agent makes for a shopper. Protocols such as the Agentic Commerce Protocol, the Universal Commerce Protocol and AP2 aim to make that safe for buyer, store and payment provider. See agentic commerce protocols explained.

Measuring it

Agent readiness. How well AI agents can get into your site, understand it, find their way around it and complete tasks on it. See what is agent readiness?

AgentScore. Our free score out of 100 for agent readiness, across Access (25 points), Readability (25), Navigability (20) and Task completion (30). The free scan marks Task completion “not tested” and scores the other three. See how AgentScore works.

Check. One thing AgentScore tests, such as “Buttons and links have names”, reported as a pass, warning or fail, with who usually fixes it.

Ghost Agent. A real AI agent we send through a journey on your site, on a schedule, to see whether agents can complete it. You get a step-by-step replay of each run, and an alert when a journey breaks. See Ghost Agent.

Run and pass rate. A run is one attempt by one agent at a journey. AI agents don't behave the same way every time, so journeys are tried several times, and the pass rate is the share of runs that succeed.

Flaky. A journey where some runs pass and some fail. Agents can do it, but not reliably, which usually points to a slow page, a pop-up that appears sometimes, or an unclear control.

Agent traffic. Visits from AI agents of every kind. Most never run your analytics tag, so it's measured from server or CDN logs. See how to measure agent traffic.

AI referral. A person who arrives at your site by clicking a link in an AI assistant's answer. Unlike agent visits, these do show up in analytics. See tracking visits and sales from AI assistants.

Crawls per AI referral. How many pages an AI company's agents fetched for every visitor its assistant sent you. A low number means you get traffic back for what they read.

To see where your own site stands on each of these, run AgentScore. It's free and takes under a minute.