Learning Center · Learning paths · Marketing, SEO and IT

Let the right agents in

robots.txt, bot protection, CDN rules and verification, so assistants get in and impostors don't.

6 lessons · about 39 minutes

  1. robots.txt for AI agents: who to allow and who to blockThe three kinds of AI agent, the robots.txt names each one uses, a sensible default, and the most-specific-group rule that catches most people out.7 min · Guide
  2. Bot protection and CAPTCHAs: stop blocking the agents you wantWhy bot protection catches AI assistants, how to tell if it's happening, and how to tune your CDN, WAF and CAPTCHAs so good agents get through and bad bots don't.7 min · Guide
  3. Setting up your CDN for AI agentsHow to set up your CDN and WAF for AI agents: verified bot categories, signed requests, rule order, challenge versus block, and logging.6 min · Guide
  4. How to tell if an AI crawler is realAnyone can claim to be GPTBot or Googlebot. How to verify AI agents with published IP ranges, reverse DNS and signed requests, and what to do when there's no proof.7 min · Guide
  5. Rate limits for AI agents: fair limits that don't turn customers awayHow to set rate limits that stop scrapers but not AI assistants: 429 with Retry-After, per-agent limits for verified agents, and what to watch.5 min · Guide
  6. How to write an llms.txt fileWhat llms.txt is, what to put in it, a complete example for an online store, and the mistakes that make it useless to AI agents.7 min · Guide

By the end: You know which AI agents to allow, and how to check your site isn't turning them away.

Pages you've read are marked in this browser only. Nothing is sent to us.

Start with lesson 1