robots.txt analyzer for AI.
See exactly how GPTBot, ClaudeBot, PerplexityBot, and 13 other bots interpret your robots.txt directives.
Every directive, mapped to the bot that reads it.
This tool parses every directive in your robots.txt, maps each to 16 known bots, validates syntax, and scores your configuration across 5 categories. You get a 0-100 score, per-bot status table, and suggested additions for missing AI bot rules.
What we analyze.
5 checks, each with a plain-English reason it matters. Every one maps to a line in the report.
AI Bot Coverage
Do GPTBot, ClaudeBot, PerplexityBot, and other AI bots have explicit rules? Are they allowed or blocked?
Search Bot Coverage
Are Googlebot, Bingbot, and Applebot configured? Is there a wildcard (*) fallback policy?
File Structure
Does the file exist? Sitemap directives, syntax validation, formatting quality, and file size.
AI Policy Clarity
Is there a clear distinction between browse bots (allowed) and training bots (blocked)? Deliberate policy detection.
Best Practices
Crawl-delay usage, conflicting rules, specific path rules, and file accessibility.
How We Score (100 Points Total)
Your robots.txt score measures how well your directives handle AI bots, search engines, and best practices. Each of the 5 categories contributes to a weighted total.
Want the full picture across four AI engines?
Radar orchestrates 13 AI visibility audits in staged batches: crawl access, llms.txt, structured data, brand citations, and more. It then surfaces cross-tool conflicts and ships a prioritized action plan for your domain.
Run it on your site. See what AI can read.
Free, no account, result on this page. When you want the whole picture, Radar runs all 13 audits in one pass.