Verification

Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature. See how to tell if an AI crawler is real.

Why it matters

Verification lets you treat agents by who they really are, not who they say they are. Without it you have to choose between letting impostors in and turning real assistants away. With it you can welcome the agents you want and block the rest, and trust that your agent traffic numbers describe real agents.

Read the guide How to tell if an AI crawler is real Anyone can claim to be GPTBot or Googlebot. How to verify AI agents with published IP ranges, reverse DNS and signed requests, and what to do when there's no proof. Read it

More terms in getting in

• robots.txt A file at the root of your site that tells crawlers and agents which pages they may visit.
• Bot protection Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors.
• CDN Content delivery network: the service that sits in front of your site and delivers it quickly around the world.
• CAPTCHA or challenge A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page.
• Rate limit A cap on how many requests a visitor can make in a given time.
• Impostor (spoofed agent) A request that uses a trusted agent's name but fails verification.

Every term, A to Z → · Read the glossary in full →