Diffbot User-Agent & Blocking Rules
Operated by Diffbot
Scrapers Caution
What is Diffbot?
Diffbot uses computer vision and NLP to extract structured data from web pages.
Test if Diffbot can access your URL
Enter URL to test if Diffbot is allowed or blocked.
Is Diffbot Safe?
It is recommended to use caution with Diffbot.
This bot may have legitimate uses but can be resource-intensive or used for competitive intelligence.
For more information about our safety badges, visit our documentation.
User-Agent String used by Diffbot
DiffbotHow to Block Diffbot?
Add the following standard robots.txt rule to prevent Diffbot from accessing your site.
User-agent: Diffbot Disallow: /