Before an AI agent reads a word of your page, it reads a three-digit number: the HTTP status code. This guide covers the codes and redirects that decide whether agents trust a page, quote it, retry it later or drop it, with a table for out-of-stock and discontinued products and a five-minute audit you can run today.
The number before the page
Every response your server sends starts with a status code. People never see it. Agents act on it. A crawler that gets a 404 drops the page. One that gets a 503 comes back later. One that gets a 200 assumes the page is real, whatever it says.
- 2xx: here it is 200 means the page is real and the content can be used
- 3xx: look elsewhere 301 and 308 for good, 302 and 307 for now
- 4xx: not here 404 not found, 410 gone, 429 too many requests
- 5xx: our fault 500 broken, 503 down for now, try again later
The meanings are defined in RFC 9110, the HTTP standard, so every well-behaved client reads them the same way. Google publishes exactly how its crawlers treat each one. AI companies don't publish that level of detail, but their crawlers and fetchers speak the same HTTP, and the safe assumption is that they follow the standard.
This guide is about getting the code right. For the words on error pages and forms, see error messages AI agents can understand. For 429 and throttling, see rate limits for AI agents.
Redirects that help and redirects that hurt
A redirect tells an agent "the thing you asked for is at another address". Used well, it keeps old links working. Used carelessly, it sends agents in circles or to the wrong page.
Permanent or temporary
- 301 and 308 mean the page has moved for good. Use them when you rename a URL, merge products or move to https. Google treats them as a strong signal that the new address is the one to use.
- 302 and 307 mean "over there for now, but keep the original address". Use them for short-term moves, such as a sale page or an A/B test. Google treats them as a weak signal.
- 308 and 307 differ from 301 and 302 in one way that matters for forms and APIs: RFC 9110 says a client must not change the request method when following them, whereas with 301 and 302 clients may turn a POST into a GET for historical reasons.
The common mistake is a 302 left in place for years after a permanent move. Agents and search engines keep treating the old URL as the real one.
Chains and loops
Each redirect is another request, and clients give up after a limit. Google's crawlers follow up to 10 hops by default; AgentScore follows 5. Other agents don't publish their limits, so assume fewer is better. Chains build up quietly: http to https, then to www, then to a trailing slash, then to the renamed product.
- http://northwind.example/beans/kenya 301 to https
- https://northwind.example/beans/kenya 301 to www
- https://www.northwind.example/beans/kenya 301 to the new product name
- https://www.northwind.example/coffee/kenya-aa/ 200, after three extra requests
The fix is to redirect every old address straight to the final one, in one hop. Loops, where A redirects to B and B back to A, usually come from two rules that disagree, often one at the CDN and one in the platform.
Redirecting everything to the home page
Sending every missing or retired URL to the home page feels tidy. For an agent it's a dead end that looks like an answer: it asked about a specific product and got your home page with a 200, which says nothing about that product. It's the same problem as a soft 404 (below): a 200 that doesn't answer the question. Redirect to a genuinely equivalent page, or return 404 or 410.
JavaScript and meta refresh redirects
A redirect made in JavaScript only happens if the agent runs JavaScript, and many fetchers don't. Google's redirect guidance says to use JavaScript redirects only if server-side or meta refresh redirects aren't possible, because Google might never see one if rendering fails. An agent that reads HTML only will see whatever the page says before the script runs, which is often nothing.
Hard for agents
- Every old URL redirects to the home page
- Three or more hops to reach a page
- 302 for a move that's permanent
- Redirect made in JavaScript
Easy for agents
- Each old URL goes to its closest equivalent
- One hop, straight to the final address
- 301 or 308 for permanent moves
- Redirect sent by the server
Soft 404s and their opposite
A soft 404 is a page that says "not found" (or shows an empty search, or an empty category) while sending 200. Google's documentation says that when 2xx content suggests an error, an empty page or an error message, Search Console reports a soft 404. To an agent, a 200 means "this is real content", so a soft 404 can keep a dead product in an assistant's answers.
The opposite problem is just as costly and harder to spot: a real page that sends an error code to some visitors. The usual cause is bot protection. A browser gets the product page with 200; an assistant asking for the same URL gets 403, or a 200 challenge page with no product on it. Google says it doesn't use content from URLs that return 4xx codes (except 429), and drops them from its index over time. You can't see this from your own browser, so it often goes unnoticed. See bot protection and CAPTCHAs and finding the errors AI agents hit in your logs.
Don't use 401 or 403 to slow crawlers down. Google says 4xx codes other than 429 have no effect on its crawl rate. To ask a client to slow down, use 429 or 503 with Retry-After.
Out-of-stock, discontinued and seasonal products
Product pages come and go more than any other kind of page, and each situation has a right answer.
| Situation | Status | What the page should do |
|---|---|---|
| Temporarily out of stock | 200 | Keep the page. Say "Out of stock" in text and set availability to OutOfStock in your product data. Add a restock date if you know it. |
| Seasonal, back next year | 200 | Keep the page and say when it returns, so an agent can tell the shopper. |
| Discontinued, with a direct replacement | 301 | Redirect to the replacement, in one hop. |
| Discontinued, no replacement | 404 or 410 | Say it's no longer sold and link to similar products. RFC 9110 says 410 means gone and likely permanent; 404 says nothing about whether it's permanent. |
| Product URL that never existed | 404 | A helpful not-found page with search and categories. |
The common mistake is treating out of stock as an error, either by returning 404 or by redirecting to the category. That removes the page from agents' view just when a shopper might ask when it's back. More on product data in product data that AI shopping agents can read; for old and duplicate URLs, see canonical URLs.
Maintenance, overload and Retry-After
When your site is down on purpose or struggling, say so with 503 Service Unavailable, not with a 200 "We'll be back soon" page. RFC 9110 says a server may send a Retry-After header with a 503, giving either a number of seconds or a date.
HTTP/1.1 503 Service Unavailable
Retry-After: 1800
Content-Type: text/html
<h1>We're updating the store</h1>
<p>Back by 10:30 UTC. Orders placed before 10:00 are safe.</p>Google says 503 and other 5xx codes make its crawlers slow down temporarily and keep already-indexed pages for a while, though pages that keep failing are eventually dropped. A 200 maintenance page tells every crawler that your product pages now say "We'll be back soon". Keep maintenance short, and don't serve 503 for days.
429 Too Many Requests, defined in RFC 6585, is the code for "you, specifically, slow down", and it can carry Retry-After too. Our rate limits guide covers when to use it.
What AgentScore checks
AgentScore has no single status code check, but status codes run through several parts of the scan:
- Before scoring. It fetches your home page as a browser. If that returns 404 or a 5xx code, it reports the site as unreachable. If it returns 401 or 403 even to a browser, or 429, it stops and says so, since every check would only measure the block page. It follows up to five redirects per request, so a home page that takes more than five is reported as unreachable.
- "Bot protection lets AI agents through" requests the home page as ChatGPT-User, Claude-User, Perplexity-User and OAI-SearchBot. If an agent gets a 4xx or 5xx code, or a challenge page, while the browser gets through, the check fails. If an agent gets much less content than the browser, it warns.
- "AI agents can open your product, pricing and cart pages" fetches each key page as a browser and as ChatGPT-User. If the browser gets a page and the assistant gets a 4xx or 5xx code or a challenge, the check fails.
Sites that check where crawlers really come from are handled separately, so good bot protection isn't marked down; see why AgentScore stopped marking you down for checking who's visiting. AgentScore doesn't look for soft 404s or redirect chains (it only gives up after five hops), and it never sees your maintenance page. The audit below covers those.
A five-minute audit
You need a terminal and five URLs: your home page, a product, an out-of-stock product, an old URL you've retired, and a made-up URL.
curl -s -o /dev/null -L -w "%{http_code} %{num_redirects} %{url_effective}\n" \
https://northwind.example/beans/kenyaThat prints the final status code, how many redirects it took and where it ended up. Then check each result:
- Home and product pages: 200, no more than one redirect
- Out-of-stock product: 200, with "Out of stock" in the text
- Retired URL: one 301 to its closest equivalent, not the home page
- Made-up URL: 404, not 200 and not a redirect home
- The same results when you add
-A "Claude-User"to the command
The last line is a rough test: a request that only claims to be an assistant isn't the real thing, and well-configured bot protection may rightly treat it differently. If results differ, look at your CDN or bot protection logs for the real assistants' requests. Then run AgentScore to check your key pages from the outside.