robots.txt Analyzer for AI
See exactly how GPTBot, ClaudeBot, PerplexityBot, and 13 other bots interpret your robots.txt directives.
This tool parses every directive in your robots.txt, maps each to 16 known bots, validates syntax, and scores your configuration across 5 categories. You get a 0-100 score, per-bot status table, and suggested additions for missing AI bot rules.
What we analyze
Do GPTBot, ClaudeBot, PerplexityBot, and other AI bots have explicit rules? Are they allowed or blocked?
Are Googlebot, Bingbot, and Applebot configured? Is there a wildcard (*) fallback policy?
Does the file exist? Sitemap directives, syntax validation, formatting quality, and file size.
Is there a clear distinction between browse bots (allowed) and training bots (blocked)? Deliberate policy detection.
Crawl-delay usage, conflicting rules, specific path rules, and file accessibility.
How We Score (100 Points Total)
Your robots.txt score measures how well your directives handle AI bots, search engines, and best practices. Each of the 5 categories contributes to a weighted total.