# Questions to ask your CDN or bot protection vendor > Questions to ask your CDN or bot protection vendor about verified bots, Web Bot Auth, AI crawler controls, challenges, rule order and logging for AI agents. By Ghost Agent Lab · Guide · https://ghostagentlab.com/blog/questions-for-your-cdn-vendor/ Your CDN or bot protection vendor decides, request by request, whether an AI agent gets your product page or a challenge. Most of those decisions happen in defaults and managed rules you may never have read. Here are the questions to ask your vendor, grouped by topic, with what a useful answer sounds like. ## Why ask the vendor, not just the console Bot protection products change often. Vendors add agent categories, rename them, and ship new managed rules and one-click settings. A setting that was right last year may now block an assistant you want, or let through a scraper you don't. Your console shows what's switched on; your vendor can tell you what it actually does to each kind of agent. Before you ask, agree internally which kinds of agent you want: AI assistants that fetch pages for a person, AI search crawlers, training crawlers, browser agents. Our guides on [the AI agents that visit your website](https://ghostagentlab.com/articles/kinds-of-ai-agents/) and [whether to block or allow AI crawlers](https://ghostagentlab.com/blog/block-or-allow-ai-crawlers/) help with that decision. Then the questions below become "how do we get what we decided?", which a vendor can answer. 1. **Request arrives:** Claims to be an AI assistant 2. **Verified?** IP ranges, reverse DNS or a signature 3. **Rules run in order:** Security, blocks, allows, limits, challenges 4. ✓ **Page served:** Or a block, challenge or rate limit **What your vendor decides.** Every step is a place to ask how it treats AI agents. ## Verified bots and signed requests Anyone can send a request that says it's ChatGPT-User. What matters is whether your vendor can confirm a request really comes from the operator it names. Our guide to [telling real AI crawlers from fakes](https://ghostagentlab.com/articles/verify-ai-crawlers/) explains the methods. - **Which AI agents do you verify, and how?** A good answer names the method for each (published IP ranges, reverse DNS, signatures) and where the list is kept. - **Which categories do you put them in, and can rules use those categories?** You want one rule for "AI assistants", not a list of names you maintain yourself. - **How quickly do new agents and changed IP ranges reach your list?** Assistants launch and change often. - **Do you verify signed requests (Web Bot Auth)?** Signatures let agents running in the cloud prove who they are when their addresses change. Web Bot Auth builds on HTTP Message Signatures ([RFC 9421](https://www.rfc-editor.org/rfc/rfc9421.html)), and an [IETF working group](https://datatracker.ietf.org/wg/webbotauth/about/) is standardizing methods for authenticating automated clients, so support is still arriving. See [why we sign every request our agents send](https://ghostagentlab.com/blog/signed-requests/). Some vendors document this openly. Cloudflare, for example, describes [Web Bot Auth as a verification method](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/) for verified bots and agents, and lists [verified bot categories](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) such as AI Assistant and AI Crawler. The same page notes that since July 2026, new AI search crawlers go in its general Search category, a reminder that these categories move. Akamai's documentation says its known bots are [listed under verified bots, where you can set an action by category](https://techdocs.akamai.com/security-ctr/docs/dimensions-new). Names and options change, so ask your vendor what applies to your plan today. ## AI crawler controls Many vendors now offer a setting to block or manage "AI bots". These are useful, and they're easy to switch on without knowing exactly what they cover. - **Which agents does your AI bot setting cover?** Training crawlers, AI search crawlers and assistants that fetch pages for a person are different. Blocking training is a business choice; blocking assistants costs you customers. - **Can we treat those kinds differently?** A good answer: yes, by category or behavior, for example allow assistants and search, block training. - **Does the setting also change our robots.txt?** Some products manage robots.txt for you. You want to know before your SEO team finds rules they didn't write. - **When you add a new agent to a category, does our setting apply to it automatically?** That's convenient, and it can also block something new you wanted. - **AI assistants:** Fetch a page because a person asked - **AI search crawlers:** Index pages for AI answers - ! **Training crawlers:** Collect content to train models - ✗ **Impostors:** Claim a known name, fail verification **Four kinds, four decisions.** Ask your vendor whether its AI controls can tell them apart. ## Challenges and checkout For a person, a JavaScript challenge is a short wait. For most AI assistants, which don't run scripts, it's a dead end, and CAPTCHAs are a dead end for every agent. In our [study of 255 stores](https://ghostagentlab.com/blog/agentscore-store-study-2026/), of 193 stores whose cart or checkout we could load, 2% blocked or challenged requests identifying as AI assistants there while browsers got through. - **What does a verified AI assistant get when it hits a challenge rule?** A good answer: verified agents skip bot challenges, but security rules still apply. - **Which of our pages have stricter rules, and why?** Login and account pages often do, for good reason. Product, cart and checkout pages usually shouldn't challenge verified agents. - **Can a challenge be replaced with a rate limit for agents?** A 429 with a Retry-After header tells an agent to slow down; a challenge tells it nothing. See [rate limits for AI agents](https://ghostagentlab.com/articles/rate-limits-ai-agents/). - **Do you add a challenge during traffic spikes or sales events automatically?** You want to know before your busiest day. ## Rule order and defaults Most agent problems come from the order of rules, not a missing rule. A broad "challenge automated traffic" rule above your allow rule for verified agents means the allow rule never runs. - **In what order do managed rules, our custom rules, bot scoring and rate limits run?** Ask for the sequence in writing, for your plan. - **Does an "allow" for verified bots skip only bot checks, or all security?** You want it to skip bot challenges while attack protection still applies. - **Which managed rules are on by default for us, and which ones affect AI agents?** Defaults are where surprises live. - **Can we run a new rule in log-only mode first?** A good answer: yes, and here's how to see what it would have caught. **Allow rule never runs:** - ✗ 1. Challenge automated traffic - ✗ 2. Allow verified AI assistants **Allow rule works:** - ✓ 1. Security rules for everyone - ✓ 2. Allow verified AI assistants - ✓ 3. Rate limits - ✓ 4. Challenge what's left **Order decides the outcome.** Ask your vendor where your allow rule sits relative to its managed rules. Our guide to [setting up your CDN for AI agents](https://ghostagentlab.com/articles/cdn-settings-ai-agents/) walks through a sensible order step by step. ## Logging and proof When an assistant tells a shopper it "couldn't access" your site, someone has to find out why. That needs logs. - **Do our logs record whether a request was a verified bot, and its category?** - **Do they record the action taken and the ID of the rule that took it?** Without the rule ID, you can't fix the right rule. - **Can we stream or export full logs, or only samples?** Samples can miss rare agents. - **How long are logs kept?** Long enough to compare before and after a change. With logs connected, Ghost Agent Lab's Agent traffic page shows which agents visit and what they got back, and Verification & spoofing checks requests that claim to be known agents against the proof their operators publish. See [finding the errors AI agents hit in your logs](https://ghostagentlab.com/articles/agent-errors-in-logs/). > **Check the answers from outside:** run [AgentScore](https://ghostagentlab.com/agentscore/) before and after any change. It requests your home page as a browser and as four AI agents (ChatGPT-User, Claude-User, Perplexity-User and OAI-SearchBot), and fails the bot protection check if one is blocked or challenged while the browser gets through. Its agent requests come from our servers, not the AI companies', so a vendor that verifies properly may rightly turn them away. To tell those cases apart, AgentScore also sends a request claiming to be Googlebot: if that is turned away too, it marks the check "not tested" rather than failed. [Why our scanner wears other names](https://ghostagentlab.com/blog/scanner-wears-other-names/) explains the trade-off.