Anthropic Bots & Crawlers

Anthropic, creator of the Claude family of AI models, uses several crawlers for different purposes. ClaudeBot is the main web crawler that scrapes content to build training datasets for Claude AI models. Claude-Web and Claude-User represent user-initiated browsing when someone asks Claude to visit a specific URL to answer questions. Claude-SearchBot indexes content specifically for search-related features within the Claude ecosystem. Publishers can block ClaudeBot to prevent AI training while still allowing user-initiated browsing through Claude-Web and Claude-User. Anthropic provides clear documentation on how to control each crawler through robots.txt directives.