Bot protection

Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors. Set too strictly, they turn away AI assistants as well as scrapers. See bot protection and CAPTCHAs.

Why it matters

Bot protection is the most common reason AI assistants are turned away from sites that want them. In our study of 255 online stores, 13% blocked or challenged AI assistants somewhere. The fix is usually a settings change, not a rebuild: allow the agents you want, verified, and keep blocking the rest. It's normally a job for your security or CDN team.

Read the guide Bot protection and CAPTCHAs: stop blocking the agents you want Why bot protection catches AI assistants, how to tell if it's happening, and how to tune your CDN, WAF and CAPTCHAs so good agents get through and bad bots don't. Read it

More terms in getting in

• robots.txt A file at the root of your site that tells crawlers and agents which pages they may visit.
• CDN Content delivery network: the service that sits in front of your site and delivers it quickly around the world.
• CAPTCHA or challenge A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page.
• Rate limit A cap on how many requests a visitor can make in a given time.
• Verification Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature.
• Impostor (spoofed agent) A request that uses a trusted agent's name but fails verification.

Every term, A to Z → · Read the glossary in full →