Agent directory · By company

Yandex's AI agents and crawlers

Yandex runs 3 agents that visit websites: 1 AI and search crawler, 2 AI training crawlers. Here's what each one does, and how to let them in or keep them out.

AI and search crawlers

Indexes pages so a search engine or AI search tool can find, cite and link to them. Allow it, unless you deliberately want to stay out of that search engine. Blocking it removes you from its results and answers.

YandexBot: Yandex's crawler for its search engine.

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

YandexAdditional
YandexAdditionalBot

Robots.txt for every Yandex agent

To keep Yandex's training crawlers out but still let its assistants, search and agents in:

User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Disallow: /

To block every Yandex agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: YandexBot
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really Yandex

Yandex publishes ways to check that a request really came from it:

Reverse DNS: names ending in yandex.ru, yandex.net, yandex.com

Requests that use Yandex's names but fail these checks are impostors. Ghost Agent Lab checks every request this way. How to verify AI crawlers.

Which Yandex agents visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from them: what they read, what your site sent back, and whether each one was genuine. See agent traffic, or check whether AI agents can use your site for free.