Rate limits for AI agents: fair limits that don't turn customers away

Rate limits protect your site from scrapers and floods of automated traffic. Set badly, they also turn away the AI assistants that are looking up your products for a customer. This guide explains how to set limits that stop abuse without blocking the agents you want, and how to say "slow down" in a way agents understand.

Why limits catch good agents

Most rate limits count requests per IP address. That made sense when one address meant one visitor. It works less well for AI agents:

• Agents share addresses. Assistant fetchers and browser agents run in cloud data centers, so many different people's requests can arrive from a small pool of addresses. A per-IP limit treats them as one heavy user.
• Agent visits come in bursts. A person asks an assistant to compare three jackets, and it opens a category page, three product pages and a returns policy within seconds. That's a normal visit, but it can look like a spike.
• Crawlers and assistants get lumped together. A training crawler reading your whole catalog and an assistant answering one shopper's question have very different value to you, but a single "bots" limit treats them the same.
• Blocks look like limits. Many setups answer a rate-limited request with a 403 Forbidden or a challenge page. The agent can't tell it should wait, so it reports your site as unavailable.

Say "slow down" properly: 429 and Retry-After

HTTP has a status code made for this. 429 Too Many Requests, defined in RFC 6585, tells the client it has sent too many requests. Pair it with a Retry-After header, defined in RFC 9110, saying how many seconds to wait:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: text/plain

Too many requests from this client. Please wait 30 seconds and try again.

Well-behaved crawlers and agents treat this as an instruction, not a door slammed shut. Google's crawler documentation, for example, says Googlebot slows down when it gets 429, 500 or 503 responses. Compare the alternatives:

Response	What the agent concludes
429 with Retry-After	Wait, then try again. The site is fine.
429 without Retry-After	Back off, but guess for how long.
503 Service Unavailable	The site is having problems. Fine for real outages, misleading for rate limits.
403 Forbidden	I'm not allowed here. Many agents give up and say so.
200 with a challenge or "access denied" page	Confusing. Some agents read the challenge text as your content.

Keep the 429 lightweight: a short plain-text message, no heavy page, and never a CAPTCHA, which agents can't solve. There's also an IETF draft for RateLimit headers that tell clients their remaining allowance before they hit the limit. It's still a draft, so treat it as a bonus for API clients, not a replacement for Retry-After.

Limit by who, not just by address

The fairest limits are set per agent, based on verified identity. Most CDNs and bot management products can verify the major AI agents against their operators' published IP ranges, reverse DNS or signed requests (see how to tell if an AI crawler is real). Once you know who's asking, you can give each kind of visitor a limit that fits its job:

Visitor	What it's doing	Suggested approach
Verified AI assistants (such as ChatGPT-User, Claude-User, Perplexity-User)	Fetching pages because a person asked	Generous limits that allow short bursts. Each request stands in for a customer.
Verified AI search crawlers (such as OAI-SearchBot)	Indexing pages so you appear in AI answers	A steady crawl rate. Use 429 with Retry-After to pace them, not blocks.
Verified AI training crawlers (such as GPTBot)	Collecting content for model training	Your business decision. Pace them, or opt out in robots.txt. See robots.txt for AI agents.
Unverified traffic claiming an AI agent's name	Unknown, sometimes impersonation	Normal or stricter limits. Never give extra allowance to a name alone.
Everything else	People, browsers, other bots	Your existing limits, tuned so a busy shopper never hits them.

The right numbers depend on your traffic and infrastructure, so we don't suggest specific figures. Start from what your logs show a normal agent visit looks like, then set limits comfortably above it.

Limit actions more than pages

Abuse mostly targets actions: logins, account sign-ups, discount codes, gift card balances, stock checks and search. Pages that only display information are cheap to serve, especially from a cache. Put your tightest limits on:

• Login, sign-up and password reset
• Coupon, gift card and checkout submission endpoints
• Search and filter endpoints that hit your database
• Public APIs and any MCP server you offer

Keep product, category, pricing, policy and help pages loose, and cache them well. Those are the pages agents need to answer questions about you, and the ones that turn into sales.

Other ways to reduce load

• Cache aggressively. A product page served from a CDN cache costs almost nothing, however often it's requested.
• Support conditional requests. ETag and Last-Modified headers let crawlers ask "has this changed?" and get a tiny 304 Not Modified response if not.
• Keep your sitemap accurate. Correct lastmod dates help crawlers revisit only what changed. See XML sitemaps for AI agents.
• Don't rely on Crawl-delay. This robots.txt line is ignored by Google and supported unevenly elsewhere. Rate limits with 429 work for every client.

How to tell if limits are hurting you

1. Check your logs by agent. Look at the share of requests from AI assistants that get 429, 403 or challenge responses. A handful of 429s for a crawler is healthy; repeated 429s or 403s for assistant fetchers mean customers' questions are going unanswered. See how to measure AI agent traffic.
2. Run AgentScore. It requests your home page as a normal browser and as several AI agents, and requests your key pages, cart and checkout as AI assistants. Any error status an agent gets while the browser gets through, including 429, counts as blocked in the bot protection, key pages and checkout checks.
3. Ask your CDN provider whether its rate limiting treats verified bots separately, and whether it returns 429 or 403 when a limit is hit.

Rate limits and bot protection work together. For the wider picture, read bot protection and CAPTCHAs and, for the business decision about which agents to let in at all, should you block AI crawlers?