Changelog, New Features and Updates

Complete version history and detailed changelog for all CrawlerCheck releases.

v1.0.0

v1.1.0

v1.1.1

v1.2.0

v1.3.0

v1.3.1

v1.3.2

v1.3.3

v1.4.0

v1.5.0

What's New in v1.5.0 - December 05, 2025

NEW: Crawler Directory: Launched a searchable database of 150+ web crawlers, AI bots, and SEO tools with real-time filtering.
Bulk Robots.txt Generator: Instantly generate "Allow" or "Block" rules for all currently filtered bots in one click—perfect for creating custom blocklists.
Directory-Wide URL Checker: Check if a specific URL is blocked by any of the currently viewed bots simultaneously.
Detailed Bot Profiles: Every bot now has a dedicated page featuring its owner, purpose, safety verdict, and specific user-agent strings.
Dynamic Filtering: Filter bots by Category (AI, Search, SEO) and Safety Rating (Safe, Aggressive, Malicious) to find exactly what you need.
Responsive & Accessible: A fully responsive layout with optimized mobile views and improved accessibility for screen readers.

What's New in v1.4.0 - December 04, 2025

The "Deep Inspection" Update: A major architectural upgrade bringing the engine in line with strict modern bot standards.
Source-Specific Blocking: The tool now distinguishes between blocks originating from HTTP Headers (HeaderNoIndex) versus HTML Meta Tags (MetaNoIndex) for faster debugging.
Crawl Budget Indicators: Added visual badges to instantly highlight categories where bots are allowed to visit but instructed not to index—a primary source of crawl budget waste.
RFC 9309 Strict Compliance: The decision engine has been rewritten to perfectly mimic the complex "Longest Match" and priority logic used by Googlebot and GPTBot.
Expanded Crawler Database: Added support for 50+ new User-Agents, with a specific focus on the latest generation of AI Crawlers (Anthropic, OpenAI, Perplexity).
Deep Header Analysis: The engine now processes multi-layered response headers to catch edge-case directives often missed by standard SEO tools.
Advanced Security: Implemented a new custom network layer with hardened SSRF protection and improved stability.

What's New in v1.3.3 - November 29, 2025

Privacy Policy and Cookie Policy: Added privacy policy and cookie policy pages to comply with GDPR and CCPA regulations.
Security: Added security measures to protect the website from attacks.
Performance: Improved performance of the website.
Code quality: Improved code quality of the website.
All previous features and security measures remain unchanged.

What's New in v1.3.2 - July 22, 2025

Dark theme support: Added beautiful dark mode with automatic system preference detection and manual toggle option.
Performance optimizations: Enhanced caching strategy with improved service worker and Nginx configuration for faster page loads.
Better user experience: Improved preloading behavior for smoother navigation.
Offline support improvements: Enhanced service worker with better error handling and offline fallback pages.
Code quality improvements: Centralized version management system for more reliable deployments.
All previous features and security measures remain unchanged.

What's New in v1.3.1 - July 19, 2024

What's New in v1.3.0 - July 18, 2024

Security and stability improvements: Enhanced backend protection against abuse and malicious requests.
Rate limiting: Added safeguards to prevent excessive or automated use, ensuring fair access for all users.
All previous features and user experience remain unchanged for normal use.

What's New in v1.2.0 - July 17, 2024

User-agent categorization system: Results are now organized into 8 distinct categories (Search Engines, AI Bots, SEO Tools, Social Bots, etc.) for easier analysis.
Enhanced crawler detection: Added 20+ new user agents including the latest AI bots (MistralAI, Perplexity), SEO tools (SearchmetricsBot, Screaming Frog), and Yahoo crawlers.
URL sharing and bookmarking: Results can now be shared via direct links with query parameters.
Improved results navigation: Category sections now use SEO-friendly anchors for better organization.
Performance improvements: Faster response times and better handling of large robots.txt files.
Deeper crawler analysis: Better detection of crawl delays and user-agent specific rules.
Improved error handling: More detailed feedback when issues occur during analysis.
Updated API response structure: Results now include categorized data for better integration with other tools.
Enhanced caching system: More efficient result storage and retrieval.

What's New in v1.1.1 - July 8, 2025

Backend HTTP client now mimics real browsers more closely for improved compatibility with strict sites.
Automatic cookie handling and support for HTTP/2 enabled.
Referer header now transparently set to https://crawlercheck.com for all requests.
Improved diagnostics: HTTP status code and response body are logged for non-2xx responses to help troubleshoot blocks.
All previous features and compatibility with simpler sites are preserved.

What's New in v1.1.0 - July 8, 2025

Robots.txt parser now fully supports group precedence and ignores duplicate user-agent groups per standard.
Wildcard and end-anchor ($) rules are handled per RFC and Googlebot behavior.
Query string rules (e.g., Disallow: /*?SID=) now match URLs with query parameters.
Improved error handling and user feedback in both backend and frontend.
Accessibility and mobile responsiveness improvements.
Production logging and error reporting polish.

Going Live v1.0.0 - July 8, 2025

Initial release of CrawlerCheck with crawler detection and robots.txt parsing functionality.
Core version of CrawlerCheck is v1.0.0

If you would like to suggest a new feature or report a bug, please use our official X (Twitter) account: @crawlercheck

Featured & Supported

We are proud to be featured on major platforms! Support CrawlerCheck by checking out our listings below and helping us spread the word.