Learning Center · What's going wrong?
AI assistants can't get into my site
Blocked by robots.txt, bot protection, location rules or rate limits.
How to confirm it
A free AgentScore scan runs these 5 checks that catch it. A warning or failure on any of them points here.
- robots.txt lets AI assistants and search agents in If robots.txt shuts out AI assistants and AI search, they can't read your pages, so they leave you out of answers and recommendations.
- Bot protection lets AI agents through When bot protection blocks AI agents, a person asking ChatGPT or Perplexity about you gets nothing back, or gets a competitor instead. Our test requests come from our servers, not the AI companies', so we also send one claiming to be Googlebot. If that's turned away too, your protection is checking where requests really come from, which is the right setup: we can't tell whether the real agents get through, so the check is skipped and never counts against you. A request that gets no answer within 15 seconds isn't counted as a block either; if every one times out, the check is skipped, and it's worth looking at your page speed.
- No CAPTCHA or challenge on arrival Agents can't solve CAPTCHAs. A challenge on arrival stops them before they see anything.
- AI agents can open your product, pricing and cart pages Product, pricing and cart pages are what an assistant needs to answer “how much is it?” or help someone buy. If agents can't open them, the sale goes elsewhere. Each page is checked against the robots.txt of the address it's on, so www and non-www rules both count. If your bot protection checks where requests really come from, our test requests can't show whether real assistants get through, so a block isn't counted. A page that takes more than 15 seconds to answer our assistant request is left untested rather than counted as blocked.
- AI agents aren't blocked or shown a CAPTCHA at the cart and checkout A CAPTCHA or block at cart or checkout stops an agent at the last step, after it has done all the work, so the order is lost. If your bot protection checks where requests really come from, our test requests can't show whether real assistants get through, so a block isn't counted.
Step-by-step fixes
- Let AI assistants in through robots.txt on Shopify
- Let AI assistants in through robots.txt on WooCommerce
The guides
- robots.txt for AI agents: who to allow and who to block The three kinds of AI agent, the robots.txt names each one uses, a sensible default, and the most-specific-group rule that catches most people out.
- Bot protection and CAPTCHAs: stop blocking the agents you want Why bot protection catches AI assistants, how to tell if it's happening, and how to tune your CDN, WAF and CAPTCHAs so good agents get through and bad bots don't.
- Setting up your CDN for AI agents How to set up your CDN and WAF for AI agents: verified bot categories, signed requests, rule order, challenge versus block, and logging.
- How to tell if an AI crawler is real Anyone can claim to be GPTBot or Googlebot. How to verify AI agents with published IP ranges, reverse DNS and signed requests, and what to do when there's no proof.
- Geo-blocking, VPN blocks and AI agents Why country, VPN and data center blocks and IP-based redirects catch AI agents visiting from cloud servers abroad, and how to let verified agents through.
- Rate limits for AI agents: fair limits that don't turn customers away How to set rate limits that stop scrapers but not AI assistants: 429 with Retry-After, per-agent limits for verified agents, and what to watch.
- Age gates and region pickers that stop AI agents How age checks, country pickers and language splash pages stop AI agents, and how to keep the gate you need while agents can still read products.
- Status codes, redirects and soft 404s for AI agents The HTTP status codes and redirects that decide whether AI agents trust, retry or drop your pages, plus what to send for out-of-stock products.
Not your problem? Pick another.