Your next customer won't be human
For twenty-five years, websites have been built for two kinds of visitor: people, and the search crawlers that send people. A third kind is arriving fast. AI agents now read, compare, book and buy on people's behalf, and most websites can't see them, don't know whether they succeed, and were never designed for them.
A new kind of visitor
When someone asks an AI assistant "which of these running shoes is best for flat feet, and is it in stock in a 10?", a person never visits your site. Software does. That software comes in three broad kinds:
- Training crawlers, such as GPTBot and ClaudeBot, collect pages to teach AI models. They shape what AI knows about you over months.
- AI search and assistant fetchers, such as OAI-SearchBot, ChatGPT-User and Perplexity-User, fetch your pages in the moment to answer a question. They decide whether you're the answer today.
- Browser agents drive a real browser for a person: searching your catalog, filling in forms, adding to cart, and in some cases checking out.
The last two are the ones that matter most to revenue, because each visit stands in for a real customer with a real intent. If the agent can't find the price, read the return policy, or get past a pop-up, the customer doesn't get a worse experience on your site. They get a recommendation for someone else's.
Why nobody noticed
Most websites have no idea how many agents visit, for a simple reason: analytics tools count visitors with a JavaScript tag, and most agents never run it. Crawlers and assistant fetchers request the HTML and leave. Your dashboards show a quiet day while your server logs show a busy one.
The agents that do run JavaScript often look like ordinary browsers, so they blend into the human numbers. Either way, the question "did the agent get what it came for?" goes unanswered.
What goes wrong
When we test sites with real agents, the failures fall into the same few patterns:
- They're turned away at the door. A robots.txt rule written years ago, or bot protection that challenges anything that isn't a person, blocks the assistants that were trying to send customers.
- They can't read the page. Prices and product details only appear after JavaScript runs, and there's no structured data to fall back on.
- They can't find their way. Unlabelled icon buttons, menus that only open on hover, and cookie banners that cover the page stop an agent the way a locked door would.
- They can't finish the job. The agent finds the product but can't choose a size, or reaches the cart but can't find the checkout button.
None of these show up in a normal QA pass, because a person gets through every one of them without noticing.
The gap in today's tools
Two kinds of tools touch this problem, and neither solves it.
Bot management decides who to block. It's essential, and it's built to keep bad traffic out, which means good agents are often collateral damage. AI visibility tools track what chatbots say about your brand. That's useful too, but it measures the answer, not whether an agent could use your site to get there.
What's missing is the thing we've always done for human visitors: test the experience. Can an agent actually complete the journeys that make you money, and does it still work after this week's release?
Agent experience
We think agent experience will become a discipline of its own, next to user experience and SEO. It has three parts, and they're how we've built Ghost Agent Labs:
- Observe. See every crawler, assistant and browser agent that visits, from server and edge data rather than a JavaScript tag, and check which ones are real.
- Test. Send real AI agents through checkout, sign-up and search on a schedule, and get alerted when they fail where a person would succeed.
- Improve. Score agent readiness, fix the highest-impact problems first, and re-test to prove the fix worked.
It's the same idea behind Ghost Inspector, which has tested websites for human visitors for years, applied to the visitors that aren't human.
What to do this week
- Read your robots.txt and check you aren't blocking AI assistants or AI search by accident. Our guide to robots.txt for AI agents walks through it.
- Ask your CDN or bot protection vendor how it treats verified AI agents, and whether it can tell real ones from fakes. See how to tell if an AI crawler is real.
- Add an llms.txt file that points agents to your most important pages.
- Run AgentScore. In under a minute, it shows how AI agents are let in, what they can read and what gets in their way.
Your next million visitors won't be human. The sites that are ready for them will win the customers they bring.