Ghost Agent Lab Free AgentScore Start 30-day trial

Agent directory · AI training crawler

Bytespider

ByteDance's crawler, widely reported to collect data for its AI models. It's known for heavy crawling.

Run by
ByteDance
Type
AI training crawler
Robots.txt name
Bytespider
Verifiable
No

What Bytespider does

ByteDance's crawler, widely reported to collect data for its AI models. It's known for heavy crawling.

As an AI training crawler, it collects public pages that may be used to train AI models. The kinds of AI agent that visit websites.

Should you allow it?

A real business choice. Lean towards blocking if your content is the product (publishing, research, courses); lean towards allowing if your pages exist to sell something else. A decision guide for blocking or allowing AI crawlers.

How to allow or block Bytespider

Bytespider reads the Bytespider group in your robots.txt. To block it everywhere:

User-agent: Bytespider
Disallow: /

To allow it everywhere, even if another rule blocks other bots:

User-agent: Bytespider
Allow: /

Robots.txt is a request, not a lock: well-behaved crawlers follow it, impostors don't. Robots.txt for AI agents.

How it identifies itself

A typical user agent:

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

How to tell if it's really Bytespider

ByteDance doesn't publish IP ranges or a reverse DNS domain for Bytespider, so a request can't be proven to come from it. Anyone can send its user agent, so treat the name alone as a claim, not proof. Ghost Agent Lab reports these requests as unverified rather than guessing.

Every ByteDance agent, and a robots.txt for all of them →

Does Bytespider visit your site? Ghost Agent Lab reads your CDN or server logs and shows every request from Bytespider and 1,566+ other known agents: what they read, what your site sent back, and whether each request was genuine. See agent traffic, or check whether AI agents can use your site for free.