- of the GPTBot blocks we could date were written in a single quarter, the last three months of 2023.
- 38%
- of sites that block Anthropic’s training crawler have no rule for its agents that answer people.
- 71%
- of sites that blocked GPTBot early still update the file. Almost everything they add is another training crawler.
- 87%
- sites wrote a rule that invites an answering agent in on purpose.
- 144
2023: sites blocked GPTBot by name
OpenAI launched GPTBot in August 2023, and the New York Times sued eight weeks later. Sites reacted fast. Of the 861 sites whose GPTBot rule we could date, 38% wrote it in the last quarter of 2023, three times as many as in any other quarter. Half wrote it within six months of the launch.
Adding a bot’s name to a text file was a reasonable answer in 2023.
2025: AI companies split their bots by job, and the lists stayed the same
Every large AI company now runs two kinds of bot. A training crawler reads pages in bulk. A second agent fetches a page when a real customer asks a question.
Most lists never added the second kind. Among sites that block Anthropic’s training crawler, 71% have no rule of any kind for its answering agents. For OpenAI it is 53%, and for Perplexity 50%. These files are not abandoned: 87% of the sites that blocked GPTBot early still update them, and almost everything they add is another training crawler’s name.
2026: some of those visits can now be charged for
Cloudflare now lets sites charge crawlers per page, and open standards are being built so that an agent can pay for a page over HTTP.
Answering agents mostly get in already. Only 351 of 5,577 readable files block every one of them. Few sites chose that on purpose: just 144 wrote a rule that invites one in. Blocking training while letting answering agents in is rare. Of the sites that block OpenAI’s training crawler, 4.0% also let its answering agents in by name. For Anthropic it is 1.9%, and for Perplexity 0.9%.
Decide by purpose, then write the names
The few sites that get this right didn’t keep longer lists than anyone else. Forbes blocks OpenAI’s training crawler and lets in its search crawler by name. It decided per purpose first. Four steps get you there.
- 1Read what your file says today. Sort its rules into three groups: training crawlers, AI search, and agents that fetch a page because a person asked. Then compare each group with what you want from it.
- 2Decide by purpose rather than by name. Agents that answer people bring citations and customers. Training crawlers take the content once.
- 3Decide what to do about crawlers that will pay. Treat paying as one more purpose your rules need an answer for: let them in, charge them, or turn them away.
- 4Check who is asking. A name in a file can’t tell a real ChatGPT visit from a program that copies its User-Agent. A signed identity can.
# Training crawlers: your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
# AI search: their citations send readers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /
# Agents fetching a page because a person asked
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /| Who is asking | Answer |
|---|---|
| Search engines that cite your pages and send readers | Let in |
| Agents fetching a page because a person asked | Let in |
| Crawlers collecting training data | Your call |
| Crawlers that offer to pay for the page | Your call |
| A famous bot’s name from the wrong network | Stopped |