Rate limit

A cap on how many requests a visitor can make in a given time. Fair limits stop scrapers without turning away agents. See rate limits for AI agents.

Why it matters

Rate limits protect your site from being overloaded, but a limit set too tight turns away agents that are doing legitimate research for customers. A fair limit tells an agent when to come back instead of refusing it outright, and gives verified agents room to work. Your developers or CDN team can usually adjust this in an afternoon.

Read the guide Rate limits for AI agents: fair limits that don't turn customers away How to set rate limits that stop scrapers but not AI assistants: 429 with Retry-After, per-agent limits for verified agents, and what to watch. Read it

More terms in getting in

• robots.txt A file at the root of your site that tells crawlers and agents which pages they may visit.
• Bot protection Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors.
• CDN Content delivery network: the service that sits in front of your site and delivers it quickly around the world.
• CAPTCHA or challenge A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page.
• Verification Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature.
• Impostor (spoofed agent) A request that uses a trusted agent's name but fails verification.

Every term, A to Z → · Read the glossary in full →