magpie-crawler User-Agent & Blocking Rules
Operated by Brandwatch
Scrapers Caution
What is magpie-crawler?
Magpie-crawler is associated with Brandwatch (formerly Brandwatch / Crimson Hexagon) for social listening.
Test if magpie-crawler can access your URL
Enter URL to test if magpie-crawler is allowed or blocked.
Is magpie-crawler Safe?
It is recommended to use caution with magpie-crawler.
This bot may have legitimate uses but can be resource-intensive or used for competitive intelligence.
For more information about our safety badges, visit our documentation.
User-Agent String used by magpie-crawler
magpie-crawlerHow to Block magpie-crawler?
Add the following standard robots.txt rule to prevent magpie-crawler from accessing your site.
User-agent: magpie-crawler Disallow: /