Free robots.txt check

See what your robots.txt tells AI agents.

Enter your domain. We read your robots.txt and show which of 15 AI bots it lets in and which it blocks, with a grade from A to F.

The check is free, and you don’t install anything.

  • nytimes.comA
  • forbes.comA
  • bbc.comA
  • cloudflare.comB
  • imdb.comC
  • weather.comC
  • reddit.comD
  • shopify.comD
  • github.comD
  • medium.comF
How ten large sites scored on July 21, 2026, on what their robots.txt declares.

The check looks for these agents.

Each kind visits for a different reason, so a robots.txt can treat them differently.

71%Among top-10,000 sites that block Anthropic’s training crawler, 71% have no rule for Anthropic’s bots that answer people’s questions.Sitedex top-10,000 robots.txt study, July 2026

Agents working for a person

They fetch a page when someone asks ChatGPT, Claude or Perplexity about it.

  • ChatGPT
  • Claude
  • Perplexity

AI search

They index your pages so AI answers can cite and link to you.

  • ChatGPT search
  • Claude search

AI training crawlers

They copy pages in bulk to train models.

  • GPTBot
  • ClaudeBot
  • Anthropic AI crawler
  • PerplexityBot
  • Google AI training
  • Apple AI training
  • Meta AI crawler
  • Amazonbot
  • Bytespider
  • CCBot

The grade reads the rules a site declares in robots.txt for 15 AI bots, the same set as Sitedex’s July 2026 study of the top 10,000 sites (CC BY 4.0). It covers only what the file says. It can’t tell you what your site enforces or which agents actually visit. Read the study.

Scoring

How we score a robots.txt

Three checks add up to 100 points, and the total sets the letter. We score your site the way we scored the 10,000.

Awareness40 points
Does your file say anything about AI search, agents answering people, and training crawlers?
Consistency40 points
If you block a company’s training crawler, do you also have a rule for its agents that answer people?
Decision20 points
Did you write a rule, either way, for at least one agent that answers people?
  • A85–100
  • B70–84
  • C55–69
  • D40–54
  • FBelow 40

Grades read declared robots.txt policy only, not what a site enforces. The top-10,000 baseline is Sitedex’s July 3, 2026 scan (CC BY 4.0); grading is our own. Adding the AI training opt-out? How to opt out of AI training with robots.txt