Google Bots & Crawlers

Google operates the world's largest family of web crawlers, designed to discover, index, and analyze web content for various Google services. The main Googlebot crawler powers Google Search, while specialized variants handle images (Googlebot-Image), videos (Googlebot-Video), and news content (Googlebot-News). Google-Extended is a separate token that allows publishers to control whether their content is used for AI model training without affecting search visibility. GoogleOther handles research and development tasks. Storebot-Google verifies product pricing for Google Shopping. The Google-InspectionTool mimics Googlebot for the URL Inspection feature in Search Console. This comprehensive crawler ecosystem ensures Google can effectively index and understand the diverse content across the web while giving publishers granular control over how their content is used.