What's New in v1.3.1 - July 19, 2024
- Performance optimizations: improved backend efficiency.
- DNS caching: Added intelligent caching for faster SSRF protection checks.
- Frontend improvements: Optimized URL validation for better responsiveness.
- All security features from v1.3.0 remain active and unchanged.
What's New in v1.3.0 - July 18, 2024
- Security and stability improvements: Enhanced backend protection against abuse and malicious requests.
- Rate limiting: Added safeguards to prevent excessive or automated use, ensuring fair access for all users.
- All previous features and user experience remain unchanged for normal use.
What's New in v1.2.0 - July 17, 2024
- User-agent categorization system: Results are now organized into 8 distinct categories (Search Engines, AI Bots, SEO Tools, Social Bots, etc.) for easier analysis.
- Enhanced crawler detection: Added 20+ new user agents including the latest AI bots (MistralAI, Perplexity), SEO tools (SearchmetricsBot, Screaming Frog), and Yahoo crawlers.
- URL sharing and bookmarking: Results can now be shared via direct links with query parameters.
- Improved results navigation: Category sections now use SEO-friendly anchors for better organization.
- Performance improvements: Faster response times and better handling of large robots.txt files.
- More comprehensive analysis: Better detection of crawl delays and user-agent specific rules.
- Improved error handling: More detailed feedback when issues occur during analysis.
- Updated API response structure: Results now include categorized data for better integration with other tools.
- Enhanced caching system: More efficient result storage and retrieval.
What's New in v1.1.1
- Backend HTTP client now mimics real browsers more closely for improved compatibility with strict sites.
- Automatic cookie handling and support for HTTP/2 enabled.
- Referer header now transparently set to
https://crawlercheck.com for all requests. - Improved diagnostics: HTTP status code and response body are logged for non-2xx responses to help troubleshoot blocks.
- All previous features and compatibility with simpler sites are preserved.
What's New in v1.1.0
- Robots.txt parser now fully supports group precedence and ignores duplicate user-agent groups per standard.
- Wildcard and end-anchor ($) rules are handled per RFC and Googlebot behavior.
- Query string rules (e.g.,
Disallow: /*?SID=) now match URLs with query parameters. - Improved error handling and user feedback in both backend and frontend.
- Accessibility and mobile responsiveness improvements.
- Production logging and error reporting polish.