We graded the top 10,000 sites for AI traffic.See where yours lands.
Free · no signup · your grade is shareable.
How the web’s biggest sites scored on July 21, 2026 — declared robots.txt policy, graded by D3.
The problem
One on/off switch can’t manage 1,400+ agents.
robots.txt was built to wave crawlers in or out. Today thousands of AI agents hit the same door: some work for your customers, some train on your content, and some lie about who they are. One switch can’t tell them apart.
57.5%
Already automated
Of HTML requests are made by machines, not people — the traffic robots.txt was never built to tell apart.
Cloudflare Radar AI Insights
8×
Growing faster
AI agent traffic is growing roughly eight times faster than human traffic.
HUMAN Security, 2026
4,580:1
Taken vs. sent back
Anthropic’s crawl-to-referral ratio. Google’s is 5:1. Training crawlers take far more than they send back.
Cloudflare Radar, June 2026
Answer engines can only cite what their agents can read. If you can’t see and govern those agents, you’re invisible in the answers your buyers get.
Who’s knocking
1,400+ agents. Not all of them tell the truth.
We track every major agent across every registry. A growing share — including open-source and overseas models — publishes nothing you can verify. An on/off switch lets them in the same as anyone. We don’t punish anonymity; we expose false claims.
The study
What we read, and what we found.
In July 2026 we read the robots.txt of the 10,000 busiest sites — 15 AI bots across 8 vendors, three purposes (answers, search, training). 5,577 had a readable file. Most had no rule for the agents answering their customers.
38%
Written in one quarter
Of datable GPTBot blocks landed in a single quarter of 2023, right after the NYT–OpenAI suit.
of 861 datable blocks
71%
Half-covered
Block a vendor’s training crawler but have no rule for its answer-time agent. For OpenAI it’s 53%; for Perplexity, 50%.
of vendors’ training-blockers
351
Fully decided
Of 5,577 readable files, just 351 block every answer-time bot. For nearly everyone else, the door is open because nobody ever decided.
of 5,577 readable files
How grading works
Three questions, one grade.
We grade every site the same way we graded the 10,000 — across every tracked bot and purpose. Three questions decide the letter.
Awareness
Which agent purposes your file addresses at all — answers, search, or training.
Consistency
Whether your rules hold up across every vendor, or leak through the gaps between a crawler and its siblings.
Decision
Whether you made a real call on the answer-time agents — the ones that send you readers and customers.
The resulting grade
Now grade yours.
Enter your domain. Get your grade and the evidence in seconds — no email, no sales call.
Free · no signup · results are shareable.
The report
Read the full study.
Methodology, the three-era timeline, and the case for matching on purpose instead of names.
Data: Sitedex’s top-10,000 robots.txt scan (July 3, 2026), CC BY 4.0 — the Tranco top 10,000 domains’ declared robots.txt policy, dated via Internet Archive first-observations. Denominators: 5,577 readable files; 1,197 full-blockers; 861 datable GPTBot rules. Traffic figures: Cloudflare Radar AI Insights and HUMAN Security, 2026. Declared policy says nothing about what your edge enforces; silence rates are upper bounds on the blind spot. Analysis and grading here are our own.