Crawlspace User-Agent & Blocking Rules
Operated by Unknown
Scrapers Caution
What is Crawlspace?
A crawler associated with digital archiving or data preservation.
Test if Crawlspace can access your URL
Enter URL to test if Crawlspace is allowed or blocked.
Is Crawlspace Safe?
It is recommended to use caution with Crawlspace.
This bot may have legitimate uses but can be resource-intensive or used for competitive intelligence.
For more information about our safety badges, visit our documentation.
User-Agent String used by Crawlspace
CrawlspaceHow to Block Crawlspace?
Add the following standard robots.txt rule to prevent Crawlspace from accessing your site.
User-agent: Crawlspace Disallow: /