Learning Center · Answers · Access

What's the difference between robots.txt and llms.txt?

Short answer

robots.txt tells crawlers and agents which pages they may visit, and well-behaved ones follow it. llms.txt is a short Markdown guide that tells AI what your business does and where your key pages are. Use robots.txt to decide who gets in, use llms.txt to point agents at the right pages, and keep both at your domain root.

Comparison · Last checked November 12, 2026 · Based on robots.txt for AI agents: who to allow and who to block, How to write an llms.txt file

What's the difference between robots.txt and llms.txt?
robots.txtllms.txt
What it doesSays which pages crawlers and agents may visitTells AI what you do and where key pages are
Where it lives/robots.txt at your domain root/llms.txt at your domain root
FormatUser-agent groups with Allow and Disallow rulesMarkdown with a title, summary and described links
Who reads itWell-behaved crawlers; scrapers ignore itSome AI tools; not every AI company reads it
AgentScore checkrobots.txt lets AI assistants and search agents inllms.txt guide for AI
Common mistakeA leftover Disallow: / that blocks everythingServing a web page instead of plain text

In more detail

robots.txt is usually the first file an agent reads, so one leftover line can keep AI assistants off your whole site. Well-behaved crawlers follow it and scrapers ignore it, so abusive traffic has to be stopped at your CDN or firewall. A crawler follows only the most specific group that names it, so adding a group for one agent can quietly drop your other rules for it.

llms.txt is a proposed standard, first published in 2024, and not every AI company has committed to read it. It helps because a typical page is mostly navigation, scripts and layout. A good file lists twenty to forty well-described links to the pages customers ask about, such as shipping, returns and FAQ. Treat it as one part of agent readiness alongside clear pages, structured data and a sitemap.

What to do

  1. Open yourdomain.com/robots.txt and check no rule blocks AI search crawlers or assistant fetchers.
  2. If you opt out of AI training, block only the training crawlers by name.
  3. Write an llms.txt with twenty to forty described links and publish it as plain text at your domain root.
  4. Run AgentScore to confirm both checks pass.

Read more

All answers

How does your own site do?

AgentScore checks your site in about a minute and ranks what to fix first.