Common Crawl Bots & Crawlers
Our crawler directory currently tracks 1 web crawler operated by Common Crawl. Analyze the safety statuses, bot categories, and user-agent strings for every bot in this operator's fleet. Operator info last updated on .
About Common Crawl
Common Crawl operates CCBot, a non-profit crawler that scrapes the web to provide open datasets for research. While useful for academic purposes, CCBot consumes significant bandwidth and is sometimes blocked by webmasters.
1 Bot from Common Crawl
| Bot Name | User-Agent | Category | Safety |
|---|---|---|---|
| CCBot | CCBot | Scrapers | Caution |