Anthropic Bots & Crawlers
Our crawler directory currently tracks 6 web crawlers operated by Anthropic. Analyze the safety statuses, bot categories, and user-agent strings for every bot in this operator's fleet. Operator info last updated on .
About Anthropic
Anthropic, creator of the Claude family of AI models, uses several crawlers for different purposes. ClaudeBot is the main web crawler that scrapes content to build training datasets for Claude AI models. Claude-Web and Claude-User represent user-initiated browsing when someone asks Claude to visit a specific URL to answer questions. Claude-SearchBot indexes content specifically for search-related features within the Claude ecosystem. Publishers can block ClaudeBot to prevent AI training while still allowing user-initiated browsing through Claude-Web and Claude-User. Anthropic provides clear documentation on how to control each crawler through robots.txt directives.
6 Bots from Anthropic
| Bot Name | User-Agent | Category | Safety |
|---|---|---|---|
| Claude-Code | Claude-Code | AI Bots | Safe |
| Claude-SearchBot | Claude-SearchBot | AI Bots | Safe |
| Claude-User | Claude-User | AI Bots | Safe |
| Claude-Web | Claude-Web | AI Bots | Safe |
| ClaudeBot | ClaudeBot | AI Bots | Safe |
| anthropic-ai | anthropic-ai | AI Bots | Safe |