Names don’t scaleWhat 10,000 robots.txt files reveal about the answer economy
Teams that tend their robots.txt for years still miss the category that matters most. The lists learn names; the bots now differ by purpose — and since July 1, that gap carries a price.
38%
of datable GPTBot blocks were written in a single quarter — the 2023 reflex, timestamped
71%
of sites blocking Anthropic’s training crawler have no rule for its answer-time bots
87%
of 2023 blockers still maintain the file — almost always adding more training-crawler names
144
of 5,577 sites explicitly invited an answer-time bot in. The open door, chosen on purpose
Three years, one lesson
The list was never going to keep up.
Each era answered the same question — what do we do about bots? — with a longer list of names. The data shows where that breaks: the bots multiplied by purpose, and the lists only ever learned names.
2023
The reflex: block it by name
OpenAI ships GPTBot in August 2023; the New York Times sues eight weeks later. The web reacts at full speed — of the 861 sites whose GPTBot rule can be dated, 38% wrote it in the last quarter of 2023 alone, three times the next-tallest quarter, and half within six months of launch. The fix of the year was pasting a bot’s name into a text file. In 2023, that was a rational answer.
2025
Two kinds of bot, one kind of rule
Every major vendor now runs two kinds of bot: a training crawler that reads in bulk, and an answer-time fetcher that arrives when a real customer asks a real question. The lists never caught up. Among sites that block Anthropic’s training crawler, 71% have no rule of any kind for its answer-time bots; for OpenAI it’s 53%, for Perplexity 50%. And these aren’t abandoned files — 87% of the 2023 blockers still update them. Almost everything they add is another training-crawler name.
2026
The fetch worth charging for
On July 1, Cloudflare switched on pay-per-crawl pricing for exactly that answer-time fetch, and open rails like x402 (HTTP 402, machine-to-machine payment) are being built for the same moment. The door is already open: only 351 of 5,577 readable files block every answer-time bot. But open isn’t a decision — just 144 sites explicitly invited an answer-time bot in, and the deliberate posture that makes publishers money (block the training, welcome the answering) exists on fewer than 4% of blockers. The traffic that’s about to carry a price is the traffic almost nobody has a rule for.
Closing the gap
It takes a decision, not more upkeep.
The rare sites that get this right — Forbes blocks OpenAI’s training crawler and explicitly welcomes its search fetcher — didn’t out-maintain anyone. They decided per purpose instead of per name. Four steps get you there:
- 01Audit what your file actually says. Sort your rules into the three bot classes — training crawlers, answer engines, user-triggered fetchers — and compare against what you actually want from each.
- 02Decide per purpose, not per name. Answer-time fetches bring citations and customers; training crawls extract value once. Fewer than 4% of blockers treat them differently on purpose — that’s the money on the table.
- 03Account for the paying class. Some fetches now arrive with payment attached (x402, pay-per-crawl). That’s not a billing project — it’s one more purpose your policy should recognize, whether you welcome, price-gate, or decline them.
- 04Verify identity. A name in a file can’t tell a real answer bot from something wearing its user-agent. Signed agent identity separates the fetchers that pay from the scrapers that pretend.
# training crawlers — your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
# answer engines — their citations send readers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /
# user-triggered fetchers — a person asked for this page
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /This is the matching problem we built for.
D3 Edge matches on purpose, not names — so the next bot is covered the day it ships, your file stops guarding the wrong door, and a fetch that arrives paying is just one more purpose your policy already has an answer for.
Data: Sitedex, “Your robots.txt is a 2023 war memorial” (July 3, 2026), CC BY 4.0 — a scan of the Tranco top 10,000 domains’ declared robots.txt policy, dated via Internet Archive first-observations. Denominators: 5,577 readable files; 1,197 full-blockers; 861 datable GPTBot rules. Silence rates are upper bounds on the blind spot, and declared policy says nothing about edge enforcement. Analysis and conclusions here are our own.