Bytespider User-Agent & Blocking Rules
Operated by ByteDance
Scrapers Unsafe
What is Bytespider?
The aggressive web crawler operated by ByteDance (parent company of TikTok). It is known for high crawl rates and is primarily used to gather data for their Large Language Models (Doubao/Lark).
Test if Bytespider can access your URL
Enter URL to test if Bytespider is allowed or blocked.
Is Bytespider Safe?
No, Bytespider is generally considered unsafe.
This bot typically consumes server resources without providing search ranking value.
For more information about our safety badges, visit our documentation.
User-Agent String used by Bytespider
BytespiderHow to Block Bytespider?
Add the following standard robots.txt rule to prevent Bytespider from accessing your site.
User-agent: Bytespider Disallow: /