Arquivo-web-crawler User-Agent & Blocking Rules
Operated by FCT (Portugal)
Scrapers Safe
What is Arquivo-web-crawler?
The web crawler for Arquivo.pt, the Portuguese web archive, preserving the history of the Portuguese web.
Test if Arquivo-web-crawler can access your URL
Enter URL to test if Arquivo-web-crawler is allowed or blocked.
Is Arquivo-web-crawler Safe?
Yes, Arquivo-web-crawler is considered safe.
This bot is verified and generally respected by the SEO community.
For more information about our safety badges, visit our documentation.
User-Agent String used by Arquivo-web-crawler
Arquivo-web-crawlerHow to Block Arquivo-web-crawler?
Add the following standard robots.txt rule to prevent Arquivo-web-crawler from accessing your site.
User-agent: Arquivo-web-crawler Disallow: /