ByteDance's AI agents and crawlers

AI assistants

Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it. Blocking it means the assistant can't read your pages when a customer asks about you, so it answers from somewhere else.

• DoubaoBot: Fetches pages for Doubao, ByteDance's AI assistant.

AI training crawlers

Collects public pages that may be used to train AI models. A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else.

• Bytespider: ByteDance's crawler, widely reported to collect data for its AI models. It's known for heavy crawling.
• TikTokSpider

Robots.txt for every ByteDance agent

To keep ByteDance's training crawlers out but still let its assistants, search and agents in:

User-agent: Bytespider
User-agent: TikTokSpider
Disallow: /

To block every ByteDance agent that reads robots.txt. This also keeps out the ones acting for your customers and the ones that send you visitors, so think twice:

User-agent: Bytespider
User-agent: DoubaoBot
User-agent: TikTokSpider
Disallow: /

Robots.txt is a request, not a lock, and some agents that act for a person in real time may not read it. A decision guide for blocking or allowing AI crawlers.

How to tell if it's really ByteDance

Our registry doesn't verify ByteDance's agents against published IP ranges, so treat a request that uses one of their names as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.