Learning Center · Answers · Access
What is Web Bot Auth?
Short answer
Web Bot Auth is a new standard that lets an AI agent sign each request, so your site can prove who sent it. The agent publishes a public key on its own domain, and your CDN checks the signature. Ask your CDN or bot protection provider whether it verifies these signatures, and turn it on where you can.
In more detail
A user agent name proves nothing, because anyone can type it. Checking IP addresses works for big crawlers on fixed infrastructure, and badly for browser agents running in the cloud, whose addresses change constantly. With Web Bot Auth, the agent signs each request with a private key. Only the holder of that key can produce the signature, and a valid one proves the sender whatever network it came from.
Each signed request carries three extra headers, called Signature-Agent, Signature-Input and Signature, following HTTP Message Signatures (RFC 9421). The website or its CDN fetches the operator's public key once, from a well-known address on the operator's domain, and checks the signature. The proposal is being worked on at the IETF, and Cloudflare describes it as a verification method for verified bots and agents.
It's still early, so support is growing among agents and bot protection services. If your CDN can't verify signatures yet, nothing breaks, because signed requests are ordinary requests with extra headers. Where it can, treat a valid signature as verification and prefer it over IP allowlists. Verified agents can then get through a challenge page or a lighter rate limit, and you can count them accurately.
What to do
- Ask your CDN or bot protection vendor whether it verifies Web Bot Auth signatures.
- Turn on signature verification where your plan supports it.
- Treat a valid signature as verification in your bot rules.
- Prefer signatures over IP allowlists for the agents you want.
Read more
How does your own site do?
AgentScore checks your site in about a minute and ranks what to fix first.