Research by DNA ·

How to opt your website out of AI training with robots.txt

Add these two lines. OpenAI, Google and Microsoft plan to honor them as a no to AI training. Anthropic is building support.

User-Agent: __usage_ai-training
Disallow: /
Paste them at the top of robots.txt, above your named rules. From the IETF draft, October 30, 2026; none of the four has said it’s live yet.

Is the opt-out enough on its own?

89%

of top sites that block AI training also block a company that hasn’t signed.

That’s 1,191 of 1,343. The other 152 block only OpenAI, Anthropic and Google. The opt-out covers every company they block once those three honor it, though 59 of them block only some paths today. A re-cut of our July study.

User-Agent: __usage_ai-training
Disallow: /New: OpenAI, Google and Microsoft plan to honor itUser-Agent: GPTBot
Disallow: /Keep for now: covered once OpenAI reads the opt-outUser-Agent: CCBot
Disallow: /Still needed: Common Crawl hasn’t signedUser-Agent: ClaudeBot
Disallow: /Keep for now: covered once Anthropic reads itUser-Agent: Bytespider
Disallow: /Still needed: ByteDance hasn’t signedUser-Agent: Google-Extended
Disallow: /Keep for now: Google plans to cover it in 2027User-Agent: meta-externalagent
Disallow: /Still needed: Meta hasn’t signedUser-Agent: Amazonbot
Disallow: /Still needed: Amazon hasn’t signedUser-Agent: Applebot-Extended
Disallow: /Still needed: Apple hasn’t signed
robots.txt for a site that blocks the training crawlers of all eight companies we track, as 457 of the 1,343 do. The opt-out doesn’t cover five of its groups: their makers haven’t signed.

AI training preferences: when the opt-out starts working, and what it leaves on.

  1. Oct 30, 2026OpenAI, Google and Microsoft announce plans to honor it. Anthropic publishes the draft and starts building it.

  2. Nov 2026None of the four has said it’s live. Keep your named rules.

  3. Feb 1, 2027Microsoft starts honoring it.

  4. Early 2027Google aims to start, reading it as a Google-Extended disallow.

  5. No dateOpenAI says within months. Anthropic hasn’t given a date.

OpenAI, Google and Microsoft’s posts and Anthropic’s IETF draft and list post, October 30, 2026. The opt-out builds on the IETF’s AI Preferences (AIPREF) vocabulary.
Turns off
Training, for the companies that honor it. At Google it also ends Gemini answers drawn from your pages, since Google reads it as Google-Extended.
Leaves on
Google Search, ChatGPT search and Bing. Each has its own control.
Can’t do
Stop crawling, or recall pages already taken. A User-Agent: * group doesn’t count: 208 of 5,577 top sites block every bot that way and no training crawler by name.

Do AI crawlers respect robots.txt?

30%

of AI scrapes ignored sites’ robots.txt rules across TollBit’s network in late 2025.

TollBit, Q4 2025

  • ChatGPT-User ignored them most: 42% of its fetches. OpenAI says robots.txt may not apply when a person asks. TollBit, Q4 2025
  • Perplexity sent 3 to 6 million requests a day dressed as Chrome on a Mac, once its named crawlers were blocked. Cloudflare, August 2025

Price or block AI training crawlers before they reach a page.

DNA reads each request before your site answers, and decides by what the agent is there to do: price training crawlers, or block them, and let search in. Priced crawlers pay you for each page they take. A borrowed name is caught against the maker’s published keys and addresses. DNA is free to start, through an SDK or your nameservers.

  1. ClaudeBotAnthropic’s training crawlerPricedPays you $0.001 a page, or gets no page
  2. CCBotCommon Crawl, which hasn’t signedPricedPays you $0.001 a page, or gets no page
  3. GPTBotAn address OpenAI doesn’t publishStoppedA borrowed name gets no page
  4. OAI-SearchBotOpenAI’s search crawlerLet inChatGPT can still cite you
  5. GooglebotGoogle SearchLet inYou stay in Google
Example rules: price training crawlers at $0.001 a page, stop borrowed names, let search in. Crawlers pay in crypto, which reaches you as each payment settles. Stopping crawlers and borrowed names is on the paid plan.

Questions about opting out of AI training.

What did OpenAI announce about AI training preferences?

On October 30, 2026, OpenAI said it aims to honor the __usage_ai-training opt-out in robots.txt within months, as its AI training preference signal. Its GPTBot controls still apply.

What is AIPREF?

The IETF’s AI Preferences working group, which writes the shared words sites use to state how AI may use their content. The opt-out builds on its vocabulary, and is itself still an individual draft.

What about Content-Usage and Cloudflare’s Content-Signal?

Both state the same wish in other words: Content-Usage: train-ai=n comes from an AIPREF attachment draft, and Content-Signal: ai-train=no is Cloudflare’s own policy. OpenAI, Google and Microsoft plan to honor __usage_ai-training, which Anthropic’s Kevin Kelley drafted.

Will the AI training opt-out hurt my Google rankings?

No. Google plans to read it, from 2027, as a Google-Extended disallow when you don’t name Google-Extended yourself, and Google-Extended doesn’t affect Google Search.

I block every bot with User-Agent: *. Am I opted out of AI training?

Not for the opt-out. The draft says a wildcard group never counts, so add the __usage_ai-training group by name.

Do I still need my GPTBot and ClaudeBot rules?

Yes. None of the four has said the opt-out is live, and it won’t cover Common Crawl, ByteDance, Meta, Amazon or Apple unless they sign.

Does robots.txt stop AI crawlers?

Only the ones that choose to follow it. RFC 9309, the robots.txt standard, says its rules “are not a form of access authorization.”

Get started

Opt your site out of AI training.

Read our guide to blocking AI crawlers

Add the two lines above your named rules, then check your file free. Put DNA in front of your site to price or block the training crawlers it recognizes, as a dry run first.

Method and sources

  • Sites: our analysis of Sitedex’s scan of the Tranco top 10,000, July 3, 2026 (data, CC BY 4.0). 5,577 files were readable; sites that refused the scanner are left out, so counts run low. Declared rules only; the scan predates the opt-out. Our figures as CSV.
  • Training crawlers: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent and Amazonbot (Amazon: “may be used to train Amazon AI models”). DNA’s own training list today includes ClaudeBot, CCBot, Bytespider and meta-externalagent. PerplexityBot is out: Perplexity says it doesn’t feed foundation models. Microsoft has no separate training crawler to name.
  • The opt-out (__usage_ai-training): an individual IETF draft, not yet a standard. Dates from each company’s post, checked against Search Engine Journal. Google trains on what Googlebot fetches. The AIPREF working group. Other syntaxes: Content-Usage, Content-Signal.
  • Compliance: TollBit, Q4 2025, as reported February 4, 2026; OpenAI on ChatGPT-User; Cloudflare, August 4, 2025. In Cloudflare’s test ChatGPT-User did stop at a disallow. RFC 9309.
  • DNA identifies an agent by its maker’s published keys or address ranges, or by name where it has no such proof. Analysis and conclusions are our own.