Guide · Last updated November 8, 2026 · 8 min read
Should you block AI crawlers?
A decision guide for business sites
Block AI training if you want. Keep the agents working for your customers. A blanket block turns away the buyer along with the freeloader, so decide one kind of agent at a time.
Who this is for
Who should read this.
- An online store
- How to stay visible to shopping agents while limiting scrapers
- A publisher or media site
- What crawlers may take, and what to hold back for licensing
- A marketing or SEO team
- How to stay in AI answers without giving away everything
- A web or IT team
- Which rules to write, and where robots.txt stops working
What blocking costs you, by kind
Each kind of agent trades something different. Blocking a training crawler costs little today. Blocking a search crawler or an assistant costs you visibility in AI answers.
| Block this kind | Examples | What you lose | What you keep |
|---|---|---|---|
| Training crawler | GPTBot, ClaudeBot, Google-Extended, CCBot | Possible presence in future model knowledge | Control of your content, and leverage for any future licensing |
| Search crawler | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Citations in AI search answers. OpenAI says sites that opt out of OAI-SearchBot won't appear in ChatGPT search answers. | Crawl load |
| Assistant | ChatGPT-User, Claude-User, Perplexity-User | Answers that name you when a customer asks about you directly | Very little |
| Shopping agent | ChatGPT agent, Claude in Chrome, Muse | Sales the agent would have finished on your site | Nothing worth keeping |
Anthropic puts the cost plainly: blocking Claude-SearchBot or Claude-User may reduce your site's visibility in user searches.
Our research found 482 of the top 5,577 sites with a readable robots.txt block OpenAI's search crawler, which keeps them out of ChatGPT search answers.
Recommended settings by business type
Allow the agents tied to customers, decide on training, and block anything that can't prove who it is. The only real debate is training crawlers.
| Kind of agent | Online store | Publisher or media site | B2B or marketing site |
|---|---|---|---|
| Training crawler | Your call. Product pages are public and low risk. | Block until you have terms you like | Allow. You want models to know your brand. |
| Search crawler | Allow | Allow | Allow |
| Assistant | Allow | Allow, and watch volume | Allow |
| Shopping agent | Allow, and make checkout work for agents | Rarely relevant | Rarely relevant |
| Unknown or unverified bot | Block | Block | Block |
Why publishers differ. Your content is the product. Training crawlers read it and rarely send readers back. Blocking them today keeps your options open if licensing terms improve. Or charge them per page and keep the shoppers' agents free.
Why B2B sites lean open. Buyers ask AI tools to compare vendors. If models never read your site, someone else describes you.
Copy-ready robots.txt
Two setups cover most business sites. Paste one into the robots.txt file at the root of your site, then check it with the free robots.txt checker.
Profile A: open to AI, everything allowed. Best for B2B and marketing sites. No AI rules needed. Your existing file stays as it is.
Profile B: no training, stay in AI answers. Best for publishers, and stores that want to limit reuse of their content.
# AI training opt-out: OpenAI, Google and Microsoft plan to honor it User-agent: __usage_ai-training Disallow: / # Keep pages out of AI training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / # Stay in AI search and answers User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
The first group is the shared AI training opt-out. None of the four companies reads it yet, and it covers only them, so keep the named groups below it.
Allow lines for assistants (ChatGPT-User, Claude-User, Perplexity-User) are optional. They follow your default rules when not named.
Changes are not instant. OpenAI says a robots.txt update takes about 24 hours to reach its search systems.
What blocking can't do
robots.txt only works on agents that choose to follow it. Three gaps remain after you set your rules.
| Gap | What happens | What to do |
|---|---|---|
| Fakes | A scraper sends the name GPTBot, ignores your file, and copies your pages | Verify each request against the operator's published IP list or signature |
| Live requests | OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the request | Decide at your front door, where you see each request |
| IP blocks | Anthropic warns that blocking its IP addresses may not keep you opted out, because its bots then can't read your robots.txt | Use robots.txt for honest agents and verification for the rest |
The pattern holds across all three: robots.txt sets your rules for honest agents, and verification at your front door enforces them for everyone else. Tools like DNA do both from one place.
Questions
Will blocking GPTBot remove me from ChatGPT?
No, as long as OAI-SearchBot stays allowed. OpenAI treats the two independently.
Should I block every AI bot to be safe?
No. A blanket block also stops the assistants and shopping agents working for your customers. You lose answers and sales to protect pages that are already public.
Is blocking AI training permanent?
No. Remove the Disallow lines whenever you change your mind. Content already collected stays collected.
How do I know my rules work?
Run the [free robots.txt checker](/check) to see each agent's access. Then watch your traffic to see who follows the rules and who doesn't.
Sources
- Overview of OpenAI crawlers, OpenAI
- Anthropic crawlers, Anthropic
- How to opt your website out of AI training, D3 research, Nov 2026
- Web traffic nearly doubled, and blocking AI crawlers now costs sales, D3 research, Oct 2026