Questions to ask your CDN or bot protection vendor

Your CDN or bot protection vendor decides, request by request, whether an AI agent gets your product page or a challenge. Most of those decisions happen in defaults and managed rules you may never have read. Here are the questions to ask your vendor, grouped by topic, with what a useful answer sounds like.

Why ask the vendor, not just the console

Bot protection products change often. Vendors add agent categories, rename them, and ship new managed rules and one-click settings. A setting that was right last year may now block an assistant you want, or let through a scraper you don't. Your console shows what's switched on; your vendor can tell you what it actually does to each kind of agent.

Before you ask, agree internally which kinds of agent you want: AI assistants that fetch pages for a person, AI search crawlers, training crawlers, browser agents. Our guides on the AI agents that visit your website and whether to block or allow AI crawlers help with that decision. Then the questions below become "how do we get what we decided?", which a vendor can answer.

1. Request arrives Claims to be an AI assistant
2. Verified? IP ranges, reverse DNS or a signature
3. Rules run in order Security, blocks, allows, limits, challenges
4. Page served Or a block, challenge or rate limit

Verified bots and signed requests

Anyone can send a request that says it's ChatGPT-User. What matters is whether your vendor can confirm a request really comes from the operator it names. Our guide to telling real AI crawlers from fakes explains the methods.

• Which AI agents do you verify, and how? A good answer names the method for each (published IP ranges, reverse DNS, signatures) and where the list is kept.
• Which categories do you put them in, and can rules use those categories? You want one rule for "AI assistants", not a list of names you maintain yourself.
• How quickly do new agents and changed IP ranges reach your list? Assistants launch and change often.
• Do you verify signed requests (Web Bot Auth)? Signatures let agents running in the cloud prove who they are when their addresses change. Web Bot Auth builds on HTTP Message Signatures (RFC 9421), and an IETF working group is standardizing methods for authenticating automated clients, so support is still arriving. See why we sign every request our agents send.

Some vendors document this openly. Cloudflare, for example, describes Web Bot Auth as a verification method for verified bots and agents, and lists verified bot categories such as AI Assistant and AI Crawler. The same page notes that since July 2026, new AI search crawlers go in its general Search category, a reminder that these categories move. Akamai's documentation says its known bots are listed under verified bots, where you can set an action by category. Names and options change, so ask your vendor what applies to your plan today.

AI crawler controls

Many vendors now offer a setting to block or manage "AI bots". These are useful, and they're easy to switch on without knowing exactly what they cover.

• Which agents does your AI bot setting cover? Training crawlers, AI search crawlers and assistants that fetch pages for a person are different. Blocking training is a business choice; blocking assistants costs you customers.
• Can we treat those kinds differently? A good answer: yes, by category or behavior, for example allow assistants and search, block training.
• Does the setting also change our robots.txt? Some products manage robots.txt for you. You want to know before your SEO team finds rules they didn't write.
• When you add a new agent to a category, does our setting apply to it automatically? That's convenient, and it can also block something new you wanted.

• AI assistants Fetch a page because a person asked
• AI search crawlers Index pages for AI answers
• Training crawlers Collect content to train models
• Impostors Claim a known name, fail verification

Challenges and checkout

For a person, a JavaScript challenge is a short wait. For most AI assistants, which don't run scripts, it's a dead end, and CAPTCHAs are a dead end for every agent. In our study of 255 stores, of 193 stores whose cart or checkout we could load, 2% blocked or challenged requests identifying as AI assistants there while browsers got through.

• What does a verified AI assistant get when it hits a challenge rule? A good answer: verified agents skip bot challenges, but security rules still apply.
• Which of our pages have stricter rules, and why? Login and account pages often do, for good reason. Product, cart and checkout pages usually shouldn't challenge verified agents.
• Can a challenge be replaced with a rate limit for agents? A 429 with a Retry-After header tells an agent to slow down; a challenge tells it nothing. See rate limits for AI agents.
• Do you add a challenge during traffic spikes or sales events automatically? You want to know before your busiest day.

Rule order and defaults

Most agent problems come from the order of rules, not a missing rule. A broad "challenge automated traffic" rule above your allow rule for verified agents means the allow rule never runs.

• In what order do managed rules, our custom rules, bot scoring and rate limits run? Ask for the sequence in writing, for your plan.
• Does an "allow" for verified bots skip only bot checks, or all security? You want it to skip bot challenges while attack protection still applies.
• Which managed rules are on by default for us, and which ones affect AI agents? Defaults are where surprises live.
• Can we run a new rule in log-only mode first? A good answer: yes, and here's how to see what it would have caught.

Allow rule never runs

• 1. Challenge automated traffic
• 2. Allow verified AI assistants

Allow rule works

• 1. Security rules for everyone
• 2. Allow verified AI assistants
• 3. Rate limits
• 4. Challenge what's left

Our guide to setting up your CDN for AI agents walks through a sensible order step by step.

Logging and proof

When an assistant tells a shopper it "couldn't access" your site, someone has to find out why. That needs logs.

• Do our logs record whether a request was a verified bot, and its category?
• Do they record the action taken and the ID of the rule that took it? Without the rule ID, you can't fix the right rule.
• Can we stream or export full logs, or only samples? Samples can miss rare agents.
• How long are logs kept? Long enough to compare before and after a change.

With logs connected, Ghost Agent Lab's Agent traffic page shows which agents visit and what they got back, and Verification & spoofing checks requests that claim to be known agents against the proof their operators publish. See finding the errors AI agents hit in your logs.