crawler4j User-Agent & Blocking Rules
Operated by Developers (Library)
Scrapers Unsafe
What is crawler4j?
crawler4j is an open-source Java crawler. Like Scrapy, this usually indicates a developer using the default configuration.
Test if crawler4j can access your URL
Enter URL to test if crawler4j is allowed or blocked.
Is crawler4j Safe?
No, crawler4j is generally considered unsafe.
This bot typically consumes server resources without providing search ranking value.
For more information about our safety badges, visit our documentation.
User-Agent String used by crawler4j
crawler4jHow to Block crawler4j?
Add the following standard robots.txt rule to prevent crawler4j from accessing your site.
User-agent: crawler4j Disallow: /