Why your analytics can't see AI agents, and how to measure agent traffic

Ask most businesses how many AI agents visited their site last month and they can't say. It's not that the agents aren't coming. It's that the tools most teams use to count visitors were built in a way that can't see them. Here's why, and how to measure agent traffic properly.

Why analytics misses agents

Tools like Google Analytics count a visit when a small JavaScript tag runs in the visitor's browser and reports back. That works for people. It fails for AI agents for three reasons:

• Most agents never run the tag. AI crawlers and assistant fetchers download the HTML and leave without running JavaScript, so the tag never fires.
• Known bots are filtered out on purpose. Analytics tools exclude known bot traffic so it doesn't distort your human numbers. That's right for marketing reports, but it hides exactly the visits you want to see.
• Browser agents blend in. Agents that do run JavaScript often look like ordinary browsers, so they're counted as people, mixed in with everyone else.

You may see AI in your analytics as referral traffic: people who click a link in a ChatGPT or Perplexity answer and arrive on your site. That's valuable, but it's the result of agent visits, not the visits themselves. It doesn't show which agents read your pages, which they couldn't, or what they saw.

Where agent visits do show up

Every request to your site, from a person or an agent, passes through your web server or CDN, and they can log it whether or not any JavaScript runs. Each log line typically includes:

Field	What it tells you
User agent	Which agent it claims to be
IP address	Where it came from, which you need to verify it's real
Path	Which page or file it asked for
Status code	Whether it got the page (200), was blocked (403), rate limited (429) or hit a missing page (404)
Time and bytes	When it came, and how much content it received

Getting your logs

• Cloudflare: Logpush sends HTTP request logs to storage or another service. Availability depends on your plan.
• Vercel: log drains stream request logs to an endpoint you choose.
• Fastly, Akamai and Amazon CloudFront all offer access logs or real-time log streaming.
• Your own servers: Nginx and Apache write access logs by default.
• Hosted platforms such as Shopify don't usually give you raw request logs, which makes agent traffic much harder to see. Ask your platform what bot and crawler reporting it offers.

A quick look with the command line

If you have an Nginx or Apache access log in the standard "combined" format, this counts requests from some well-known AI agents:

awk -F'"' '{print $6}' access.log \
  | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|Amazonbot|meta-external[a-z]+|Bytespider|CCBot' \
  | sort | uniq -c | sort -rn

It's a useful first look, but it has real limits: it trusts the user agent, so impostors count as the real thing; it only knows the names you list; and it says nothing about whether the agents succeeded.

What to measure

1. Who's visiting. Requests by agent and by kind: training crawlers, AI search, assistant fetchers and browser agents. The mix matters more than the total. Assistant fetchers are people asking about you right now.
2. Who's real. Verify each agent against its operator's published IP ranges or reverse DNS, and separate verified, spoofed and "can't tell". See how to tell if an AI crawler is real. Without this, scrapers inflate the numbers.
3. What they're reading. The pages agents request most. Are they reaching products, pricing and policies, or stuck on the home page and old URLs?
4. Whether they're blocked. The share of agent requests that get 403, 429 or challenge responses. A rise usually means a bot protection change caught good agents. See bot protection and CAPTCHAs.
5. Errors. 404s for agents often point to old URLs that AI models learned and still request. Redirect them.
6. Trends. Week over week, by agent. New agents appear often, and a sudden drop from one usually means something on your side changed.

Privacy

Logs contain IP addresses, which are personal data for human visitors. You only need the full detail for bots and agents. Hash or drop IP addresses for human traffic, keep agent records separately, and set a retention period that matches your privacy policy.

Measuring traffic tells you who's coming. To find out whether they succeed, test the journeys that matter. See how to make checkout work for AI shopping agents.