Scrapy User-Agent & Blocking Rules
Operated by Developers (Framework)
Scrapers Unsafe
What is Scrapy?
Scrapy is a popular open-source web crawling framework. Seeing this user-agent usually means a developer hasn't set a custom name for their bot.
Test if Scrapy can access your URL
Enter URL to test if Scrapy is allowed or blocked.
Is Scrapy Safe?
No, Scrapy is generally considered unsafe.
This bot typically consumes server resources without providing search ranking value.
For more information about our safety badges, visit our documentation.
User-Agent String used by Scrapy
ScrapyHow to Block Scrapy?
Add the following standard robots.txt rule to prevent Scrapy from accessing your site.
User-agent: Scrapy Disallow: /