Why we sign every request our agents send
A user agent string is a name tag anyone can print. If AI agents are going to shop, book and sign up on people's behalf, websites need a better way to tell who's knocking. That's why every request our scanner and Ghost Agents send under their own name is cryptographically signed.
The problem with user agents
Every request a browser or bot makes carries a User-Agent header. Ours look like this:
AgentScore/1.0 (+https://ghostagentlab.com/agentscore/bot)
GhostAgent/1.0 (+https://ghostagentlab.com/ghost-agent)That's useful, and it's a courtesy we'll keep. But it proves nothing. A scraper can send GPTBot, ClaudeBot or AgentScore/1.0 just as easily as the real thing, and plenty do. So site owners face a bad choice: trust the label and let impostors in, or block the label and turn away the agents they actually want.
The traditional fix is to check where the request came from: published IP ranges, or a reverse DNS lookup that resolves back to the operator's domain. That works for big crawlers with fixed infrastructure. It works badly for browser agents running in the cloud, where IP addresses change all the time, and it puts the burden on every website to keep lists up to date.
What Web Bot Auth does instead
Web Bot Auth is a proposal, now being worked on at the IETF and supported by Cloudflare's verified bots program, that lets an agent prove who it is the same way websites prove who they are: with a public key.
- The agent operator publishes its public keys in a directory at a well-known address on its own domain. Ours is
https://app.ghostagentlab.com/.well-known/http-message-signatures-directory. - Every request carries three extra headers:
Signature-Agent(where to find the keys),Signature-Input(what was signed, when, and with which key) andSignature(the signature itself), following HTTP Message Signatures, RFC 9421. - The website or its CDN fetches the key once, checks the signature, and knows for certain the request came from the operator, whatever IP address it arrived from.
Anyone can copy a user agent. Only the holder of the private key can produce the signature.
How we do it
We sign with an Ed25519 key. Each signature covers the site's host name and our Signature-Agent header, carries a fresh random nonce, and expires after 60 seconds, so a captured signature can't be replayed against another site or reused later. The key ID is the key's standard JWK thumbprint, and the key directory itself is signed too, so nobody can swap in their own keys.
| Requests | Signed? |
|---|---|
| AgentScore requests under its own user agent: robots.txt, sitemaps, well-known files, page discovery | Yes |
| Every Ghost Agent request, including images and scripts from other hosts | Yes |
| AgentScore requests that deliberately imitate a browser or another agent, such as ChatGPT-User, to see how your site treats them | No, so your site treats them exactly as it would that visitor |
That last row matters. Part of what AgentScore measures is whether your bot protection treats AI agents differently from people. If we signed those requests, a CDN that trusts us would wave them through and the test would tell you nothing.
Why this matters beyond us
Agents are becoming customers' representatives. When someone asks an assistant to reorder printer ink or book a table, the agent that arrives at your site is acting for a real person with a real wallet. The sites that win are the ones that can say yes to those agents confidently, without opening the door to every scraper wearing a borrowed name.
Signed requests make that possible. A site can allow verified agents through a challenge page, give them a lighter rate limit, or simply count them accurately in analytics. None of that works if the identity is a string anyone can type.
If you run an agent: sign your requests. Publish a key directory, add the three headers, and register with the CDNs your users' sites sit behind. It's a small amount of work, and it's the difference between being trusted and being blocked.
If you run a website: check whether your CDN or bot protection can verify Web Bot Auth signatures, and prefer it over IP allowlists for the agents you want. Then run AgentScore to see how your site treats AI agents today.
Checking it's really us
The details of what our agents do, what they won't do, and how to opt out are on the AgentScore bot and Ghost Agent pages. We don't use fixed IP addresses yet, so if you want to allow us, please do it by signature or user agent rather than by IP.