robots.txt

A file at the root of your site that tells crawlers and agents which pages they may visit. Well-behaved agents follow it; scrapers ignore it. See robots.txt for AI agents.

Why it matters

robots.txt is usually the first thing an agent reads, so one line in it can decide whether AI assistants can visit your site at all. In our study of 255 online stores, robots.txt blocked AI assistants or AI search on only 4%, but it's an easy place for an old rule to block more than anyone intended. Review it with your SEO or content owner whenever your AI policy changes.

Read the guide robots.txt for AI agents: who to allow and who to block The three kinds of AI agent, the robots.txt names each one uses, a sensible default, and the most-specific-group rule that catches most people out. Read it

More terms in getting in

• Bot protection Services such as Cloudflare, Akamai or DataDome that block or challenge automated visitors.
• CDN Content delivery network: the service that sits in front of your site and delivers it quickly around the world.
• CAPTCHA or challenge A test that tries to tell people from bots, such as picking out images or waiting on a “checking your browser” page.
• Rate limit A cap on how many requests a visitor can make in a given time.
• Verification Checking that a visitor claiming to be a known agent really is, using the proof its operator publishes: IP ranges, reverse DNS or a signature.
• Impostor (spoofed agent) A request that uses a trusted agent's name but fails verification.

Every term, A to Z → · Read the glossary in full →