Anyone can send a request that says it's GPTBot or Googlebot. To let real AI agents in while keeping impostors out, you need to check where a request actually came from. There are three ways to do it, and the right one depends on the operator.
Short answer
How do I tell if an AI crawler is real?
Check where the request came from, because anyone can copy a crawler's name. Use the strongest proof the operator offers: a Web Bot Auth signature, then published IP ranges, then forward-confirmed reverse DNS. Ask whether your CDN already verifies bots, and treat requests you can't verify as unknown, handled with rate limits rather than a hard block.
Why user agents can't be trusted
Every request names its sender in the User-Agent header, and that name is a promise, not a proof. Scrapers routinely borrow the names of well-known crawlers, because many sites let those crawlers through without a challenge. If your bot rules trust the name alone, you're letting in anyone who copies it. If they distrust it, you're blocking real agents along with the fakes.
Verification resolves that. It answers one question: did this request really come from the company it names?
Trust the name
- Anyone who copies it gets in
- Or distrust it, and real agents are blocked too
Verify the source
- Real agents get through
- Impostors using their names don't
Method 1: published IP ranges
Several operators publish the IP addresses their crawlers use, as machine-readable JSON files. If a request claims to be one of their agents and comes from an address in the list, it's real.
| Operator | Agents covered |
|---|---|
| Googlebot, special-case crawlers and user-triggered fetchers (separate lists) | |
| Microsoft | Bingbot |
| Apple | Applebot |
| OpenAI | GPTBot, OAI-SearchBot and ChatGPT-User (one list each) |
| Perplexity | PerplexityBot and Perplexity-User (one list each) |
Each operator links its list from its crawler documentation. A few practical points:
- Refresh the lists often. Ranges change. Fetch them at least daily, and don't treat a request as fake because it's missing from a list that's days old.
- Match the list to the agent. An OpenAI address in the GPTBot list doesn't verify a request claiming to be ChatGPT-User.
- Handle IPv6. Several lists include IPv6 ranges.
Method 2: forward-confirmed reverse DNS
Some operators instead promise that their crawlers' IP addresses resolve to a host name on their own domain. The check has two steps, because reverse DNS on its own can be faked by whoever controls the IP address:
- Reverse lookup: look up the host name for the request's IP address, and check it ends in the operator's domain.
- Forward lookup: look up that host name's IP addresses, and check the original IP is among them.
-
A request arrives
User agent says Googlebot, from IP address
66.249.66.1 -
1. Reverse lookup
66.249.66.1→crawl-66-249-66-1.googlebot.comEnds in googlebot.com
-
2. Forward lookup
crawl-66-249-66-1.googlebot.com→66.249.66.1Points back to the same address
-
Verified
The request is from Google. If either step fails, it isn’t, whatever its user agent says.
Google's own example, from a terminal:
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
The name ends in googlebot.com and points back to the same address, so the request is from Google. Domains that operators document for this include:
| Agent | Host name ends in |
|---|---|
| Googlebot and other Google crawlers | googlebot.com, google.com or googleusercontent.com |
| Bingbot | search.msn.com |
| Applebot | applebot.apple.com |
| Amazonbot | crawl.amazonbot.amazon |
DNS lookups are slow compared with a page request, so cache the result for each IP address rather than checking on every hit.
Method 3: signed requests (Web Bot Auth)
The newest method doesn't depend on IP addresses at all. With Web Bot Auth, the agent signs each request with a private key and publishes the matching public key on its own domain. Your CDN or server checks the signature, and a valid one proves the sender, whatever network it came from.
That makes it the best fit for browser agents running in the cloud, whose addresses change constantly. It's still early, so check whether your CDN supports it. We explain how it works, and why we sign all of our own agents' requests, in Why we sign every request our agents send.
- Published IP ranges The address is on the operator's list
- Reverse DNS The host name resolves back to the operator
- Signed requests A signature proves the sender, from any network
When there's no proof at all
Not every operator publishes IP ranges, a reverse DNS domain, or signing keys for every agent. For those, you can't prove a request is real or fake; you can only weigh the evidence, such as how it behaves and how fast it requests pages.
Treat "can't tell" as its own answer, not as "fake". Blocking everything you can't verify will block legitimate agents from operators that simply haven't published proof yet.
Putting it together
- Identify: match the user agent to a known agent and its operator.
- Verify: use the strongest proof that operator offers: a signature, then published IP ranges, then reverse DNS.
- Decide: let verified agents you want through; challenge or block requests that claim a name but fail verification; and handle "can't tell" with ordinary rate limits rather than a hard block.
What to do with each answer
- Verified let through the agents you want
- Claims a name, fails the check challenge or block
- Can't tell ordinary rate limits, not a hard block
Your CDN may already do this. Many CDNs and bot-management products maintain lists of verified bots. Check that the AI assistants and AI search agents you want are in the allowed categories, and that "AI crawler" blocking rules aren't catching them by name.
Ghost Agent Lab does it for you. It identifies every agent in your server or edge traffic, checks it against published IP ranges and reverse DNS, and flags impostors, so you can see which agents are real before you decide what to allow. Missing or stale data is reported as "can't tell", never as "spoofed". Start 30-day trial.