Web Bot Auth

A new standard in which agents sign their requests cryptographically, so sites can prove who sent them. Our own scanner and Ghost Agents sign theirs. See why we sign every request.

Why it matters

Web Bot Auth could make telling real agents from impostors much simpler: a signed request proves who sent it, without keeping lists of IP addresses up to date. It's new, so support is still growing among agents and bot protection services. Ask your bot protection provider whether they check signatures, and plan to accept signed agents as they arrive.

Read the guide Why we sign every request our agents send Anyone can copy a user agent. Here's how Ghost Agent Lab signs its scanner and Ghost Agent traffic with Web Bot Auth, and why every well-behaved agent should. Read it

More terms in getting in

• robots.txt A file at the root of your site that tells crawlers and agents which pages they may visit.
• Bot protection Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors.
• CDN Content delivery network: the service that sits in front of your site and delivers it quickly around the world.
• CAPTCHA or challenge A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page.
• Rate limit A cap on how many requests a visitor can make in a given time.
• Verification Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature.

Every term, A to Z → · Read the glossary in full →