Agent traffic isn't bot traffic
Most websites sort their visitors into two piles: people and bots. People are customers. Bots are a cost, a risk, or noise to filter out of the reports. AI agents don't fit that split. Some of them are the closest thing your site has to a customer in the room, and treating them like scrapers means turning those customers away without knowing it.
The two-pile habit
The people-or-bots split made sense for a long time. Apart from search engine crawlers, which everyone learned to welcome, automated traffic was mostly scrapers, credential stuffers, inventory hoarders and uptime monitors. So the tools grew up around it. Bot protection scores each request on how human it looks. Analytics filters out known bots. Dashboards show "bot traffic" as one line, usually as a problem.
AI agents arrive in that system looking like bots, because technically they are. But "bot" now covers software doing very different jobs for very different reasons, and some of those jobs are done for a specific person who wants to buy something.
Five kinds of automated visitor
| Kind | Examples | Why it's there | What it's worth to you |
|---|---|---|---|
| AI assistants | ChatGPT-User, Claude-User, Perplexity-User | A person asked a question just now, and the assistant is reading your page to answer it | A live customer question about you |
| Browser agents | Agents that run a real browser to complete a task, such as Ghost Agents | A person asked it to do something: compare, book, buy | A customer, part-way through a journey |
| Search and AI search crawlers | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot | Indexing pages so they can appear in search results and AI answers later | Visibility in tomorrow's answers |
| AI training crawlers | GPTBot, ClaudeBot, CCBot, Google-Extended | Collecting content to train models | A business decision, not a customer |
| Other bots | SEO tools, uptime monitors, link previews, scrapers | Their own reasons | Mostly a cost, sometimes useful |
Google-Extended is a robots.txt token rather than a separate crawler, but it controls the same choice. Running through all five rows is a sixth problem: impostors. Scrapers often claim to be Googlebot or GPTBot to get past bot protection, so a name in a user agent proves nothing on its own.
The top two rows are the ones the two-pile habit hurts most. An assistant fetch is one person's question. If it's blocked, that person gets an answer about somebody else. A browser agent that hits a CAPTCHA at checkout is an abandoned cart with no record of why. Neither shows up in your analytics, because most agents never run the tracking script, and both look like "bots" to a system tuned to stop bots.
What goes wrong when it's all one pile
- Blocking the wrong visitors. A rule written to stop scrapers stops ChatGPT-User too, and nobody notices because nobody is measuring it. Our guide to bot protection and AI agents covers how this happens.
- Making the wrong call on "AI". Blocking training is a reasonable choice. Blocking every AI user agent to achieve it also removes you from assistants and AI search. Should you block AI crawlers? splits the decision properly.
- Reading growth as a threat. "Bot traffic is up" sounds like an attack. "Assistant fetches of our product pages are up" sounds like demand. They can be the same line on the same chart.
- Missing the failures. If agent traffic is filtered out of reporting, so are the 403s, challenge pages and missing pages agents keep hitting. The journey breaks and the dashboard stays green.
Treat agent traffic as its own channel
The fix isn't to welcome every bot. It's to stop treating them as one thing. In practice that means four habits:
- Measure it separately. Your server or CDN logs see every request, including the ones that never run JavaScript. Split them into the kinds above. How to measure agent traffic shows how.
- Verify before you trust. Check a claimed agent against its operator's published IP ranges or reverse DNS, or a request signature where the operator signs its requests with Web Bot Auth. Our guides to verifying AI crawlers and signed requests cover the methods.
- Set rules per kind, not per "bot". Let verified assistants and AI search through to product, pricing, cart and checkout pages. Decide on training crawlers as a separate business question. Rate-limit and challenge the rest, and block impostors outright. Rate limits for AI agents covers the middle ground.
- Watch outcomes, not just volume. For each kind, track what it got back: pages served, redirects, blocks and missing pages. A rise in blocked assistant requests is a lost-sales signal, and should reach the same people as a broken checkout.
One honest caveat: not every agent can be identified. Some browser agents use an ordinary browser's user agent and run from cloud addresses, so they look like people, or like a bot your protection doesn't recognize. Signed requests will help as more operators adopt them. Until then, the best evidence of how these agents fare is to run AI agents through your journeys yourself.
What it means for your reports
The most useful change is often the simplest: stop putting AI agents in the "bots" line. Give assistants, browser agents, AI search and training crawlers their own lines, alongside people, and report what each one got back. When a leader asks "is AI a threat or an opportunity for us?", that report answers with your own numbers instead of an opinion. Our guide to agent readiness KPIs suggests what to put in front of them each month, and tracking AI referral traffic covers the visits and sales that follow.
In the Ghost Agent Labs app, Agent traffic does this split from your server or CDN logs: people, search crawlers, AI training crawlers, AI assistants and autonomous agents, with each assistant's fetches grouped into sessions and a breakdown of how often agents were served, redirected, blocked or sent to missing pages. Verification separates verified agents from impostors.
Your next customer may arrive as a request with a user agent you've never looked at. It's worth knowing which pile you've put it in.