Ghost Agent Lab Free AgentScore Start 30-day trial

Topics

Bot protection and AI agents

Bot protection is the most common reason AI agents can't use a site: the firewall, CDN or CAPTCHA that stops scrapers stops ChatGPT and Claude too. Everything we have on letting the right agents through and keeping impostors out.

Start hereBot protection and CAPTCHAs: stop blocking the agents you wantWhy bot protection catches AI assistants, how to tell if it's happening, and how to tune your CDN, WAF and CAPTCHAs so good agents get through and bad bots don't.Read the guide

What AgentScore checks

  • Bot protection lets AI agents through When bot protection blocks AI agents, a person asking ChatGPT or Perplexity about you gets nothing back, or gets a competitor instead. Our test requests come from our servers, not the AI companies', so we also send one claiming to be Googlebot. If that's turned away too, your protection is checking where requests really come from, which is the right setup: we can't tell whether the real agents get through, so the check is skipped and never counts against you. A request that gets no answer within 15 seconds isn't counted as a block either; if every one times out, the check is skipped, and it's worth looking at your page speed. Usually fixed by Bot protection or CDN admin
  • No CAPTCHA or challenge on arrival Agents can't solve CAPTCHAs. A challenge on arrival stops them before they see anything. Usually fixed by Bot protection or CDN admin
  • AI agents can open your product, pricing and cart pages Product, pricing and cart pages are what an assistant needs to answer “how much is it?” or help someone buy. If agents can't open them, the sale goes elsewhere. Each page is checked against the robots.txt of the address it's on, so www and non-www rules both count. If your bot protection checks where requests really come from, our test requests can't show whether real assistants get through, so a block isn't counted. A page that takes more than 15 seconds to answer our assistant request is left untested rather than counted as blocked. Usually fixed by Bot protection or CDN admin
  • AI agents aren't blocked or shown a CAPTCHA at the cart and checkout A CAPTCHA or block at cart or checkout stops an agent at the last step, after it has done all the work, so the order is lost. If your bot protection checks where requests really come from, our test requests can't show whether real assistants get through, so a block isn't counted. Usually fixed by Bot protection or CDN admin

Guides

Rate limits for AI agents: fair limits that don't turn customers awayHow to set rate limits that stop scrapers but not AI assistants: 429 with Retry-After, per-agent limits for verified agents, and what to watch. Login walls and gated content: what AI agents can reachWhat AI agents can't reach behind logins and paywalls, what to keep public, how to hand off to the customer, and paywall structured data for publishers. How to make checkout work for AI shopping agentsA step-by-step guide to the shopping journey for AI agents: finding products, choosing options, adding to cart, checkout and the payment handoff. Sign-up, booking and lead forms AI agents can finishHow to build sign-up, booking and quote forms AI agents can complete, and how to handle email verification, CAPTCHAs, date pickers and the handoff. Carts AI agents can manage: quantities, promo codes and totalsHow to build a cart AI agents can change and check: named quantity and remove controls, usable promo code fields, clear totals and carts that last. How to tell if an AI crawler is realAnyone can claim to be GPTBot or Googlebot. How to verify AI agents with published IP ranges, reverse DNS and signed requests, and what to do when there's no proof.

From the blog

Questions to ask your CDN or bot protection vendorQuestions to ask your CDN or bot protection vendor about verified bots, Web Bot Auth, AI crawler controls, challenges, rule order and logging for AI agents. Agent traffic isn't bot trafficLumping AI agents in with scrapers hides customers. Why agent traffic needs its own categories, rules and reports, and how to start separating it. Why AgentScore stopped marking you down for checking who's visitingAgentScore's test requests use AI assistants' names but come from our servers, so sites that verify visitors rightly turned them away and lost points. How we fixed it, and two other corrections. Why we sign every request our agents sendAnyone can copy a user agent. Here's how Ghost Agent Lab signs its scanner and Ghost Agent traffic with Web Bot Auth, and why every well-behaved agent should. Why our scanner sometimes wears other agents' namesAgentScore requests a few pages as ChatGPT, Claude, Perplexity and Googlebot, unsigned, to see how your site treats them. Here's exactly what and why.

Free tools

Is this bot real?Check whether a request really came from GPTBot, Googlebot or another agent. What does ChatGPT see?See a page the way AI assistants fetch it, without JavaScript.

Terms to know

  • Bot protection Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors.
  • CDN Content delivery network: the service that sits in front of your site and delivers it quickly around the world.
  • CAPTCHA or challenge A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page.
  • Rate limit A cap on how many requests a visitor can make in a given time.
  • Verification Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature.
  • Impostor (spoofed agent) A request that uses a trusted agent's name but fails verification.
  • Forward-confirmed reverse DNS A verification check that an IP address's hostname belongs to the operator, and that the hostname points back to the same address.
  • Web Bot Auth A new standard in which agents sign their requests cryptographically, so sites can prove who sent them.