Learning Center · Answers · Access
What's the difference between robots.txt and llms.txt?
Short answer
robots.txt tells crawlers and agents which pages they may visit, and well-behaved ones follow it. llms.txt is a short Markdown guide that tells AI what your business does and where your key pages are. Use robots.txt to decide who gets in, use llms.txt to point agents at the right pages, and keep both at your domain root.
| robots.txt | llms.txt | |
|---|---|---|
| What it does | Says which pages crawlers and agents may visit | Tells AI what you do and where key pages are |
| Where it lives | /robots.txt at your domain root | /llms.txt at your domain root |
| Format | User-agent groups with Allow and Disallow rules | Markdown with a title, summary and described links |
| Who reads it | Well-behaved crawlers; scrapers ignore it | Some AI tools; not every AI company reads it |
| AgentScore check | robots.txt lets AI assistants and search agents in | llms.txt guide for AI |
| Common mistake | A leftover Disallow: / that blocks everything | Serving a web page instead of plain text |
In more detail
robots.txt is usually the first file an agent reads, so one leftover line can keep AI assistants off your whole site. Well-behaved crawlers follow it and scrapers ignore it, so abusive traffic has to be stopped at your CDN or firewall. A crawler follows only the most specific group that names it, so adding a group for one agent can quietly drop your other rules for it.
llms.txt is a proposed standard, first published in 2024, and not every AI company has committed to read it. It helps because a typical page is mostly navigation, scripts and layout. A good file lists twenty to forty well-described links to the pages customers ask about, such as shipping, returns and FAQ. Treat it as one part of agent readiness alongside clear pages, structured data and a sitemap.
What to do
- Open yourdomain.com/robots.txt and check no rule blocks AI search crawlers or assistant fetchers.
- If you opt out of AI training, block only the training crawlers by name.
- Write an llms.txt with twenty to forty described links and publish it as plain text at your domain root.
- Run AgentScore to confirm both checks pass.
Read more
How does your own site do?
AgentScore checks your site in about a minute and ranks what to fix first.