Get your site ready for AI agents

An AI agent is software that visits your site on someone else's behalf. Some agents work for AI companies, like GPTBot from OpenAI. Others work for your customers, like ChatGPT, Claude, or Muse fetching your page while a person waits for an answer.

A human visitor loads your page in a browser, sees your design, and clicks. An agent reads the page, pulls out what its client asked for, and moves on. The visit happens, but the person may never see your site.

Three questions describe any agent at your door:

• Who sent it? An AI company, a search tool, or a customer.
• What did it come to do? Train a model, build an answer, or complete a task like a purchase.
• Is it who it says it is? Any software can claim a famous name. Chapter 6 covers how to check.

Your customers stopped searching and clicking. They started asking their agents. The agent visits your site so the person doesn't have to.

Cloudflare draws the line this guide follows: agents don't respond to ads, but there is usually a person behind them with a job to get done. Serve them and they add to your business. Ignore them and they take from it.

Four kinds visit most sites. The big AI companies now run a separate agent for each job, so you can treat each job differently.

Other companies follow the same pattern. Google uses Googlebot for Search and a separate robots.txt name, Google-Extended, for Gemini training. Perplexity runs PerplexityBot for its index and Perplexity-User for live requests. Personal shopping agents like Muse act for one customer at a time. Every agent here has an entry in the glossary.

The split matters because each switch is independent. OpenAI says a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to keep its pages out of training.

Assistants and shopping agents bring customers, because a person sent them with a question or a purchase in mind. Search crawlers bring customers later, when an AI answer cites you. Training crawlers mostly take.

The mistake to avoid is treating all four the same. A blanket block turns away the buyer along with the freeloader. A blanket allow hands your content to anyone who asks, including scrapers using a famous name.

For stores, the shopping guide covers what agents need to complete a sale.

No. robots.txt is a polite request. Each agent decides whether to follow it, and the file can't tell a real agent from a fake one.

robots.txt is a text file at the root of your site that lists which bots may crawl which pages. It works well for the agents that choose to honor it. A typical setup that keeps training out and search in looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

Three gaps remain:

1. Live requests may skip it. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the request. Anthropic says Claude-User follows robots.txt. Policies differ by company.
2. Changes take time. OpenAI notes about 24 hours before a robots.txt change shows up in its search systems.
3. Anyone can lie. A scraper can send the name GPTBot and ignore your rules. robots.txt has no way to check.

robots.txt is still worth getting right. Check yours with the free robots.txt checker, then read chapter 6 for the part the file can't do. To decide what to allow, see Should you block AI crawlers?

A name proves nothing. Every request carries a label called a user agent, and any scraper can type GPTBot into it. Real agents prove who they are in one of three ways, from weakest to strongest.

Most sites stop at the name, which is why fakes work. A request that claims to be GPTBot but comes from an address outside OpenAI's list is not GPTBot. Treat it as an unknown scraper. Our research on the verifiability gap found 95.8% of 2,175 known bots publish nothing a site can check.

You don't need to run these checks by hand. Tools that sit at your site's front door, including DNA, check each request against the operator's list or signature before your rules apply.

Most analytics tools count people running a script in a browser. Most agents never run that script, and the tools drop the known ones on purpose.

Google Analytics 4 removes traffic from known bots automatically, using Google's research and the IAB bots list. Google says you can't turn that off, and you can't see how much it removed (Google). That keeps reports clean for humans. It also hides your fastest-growing audience.

The measurement guide walks through each row step by step, and our research puts numbers on what GA4 misses.

Three steps, in this order. None of them changes anything for your customers.

1. Check your robots.txt. See which agents you let in today, on purpose or by accident. The free checker reads your file and lists each AI agent's access.
2. Watch first, change nothing. Learn who arrives, how often, and whether each agent is real. A week of data beats a month of guessing.
3. Decide by kind. Let in the agents working for your customers. Choose what training crawlers get. Stop anything that can't prove who it is.

Then revisit monthly. New agents launch every few weeks, and each company's rules change.

• Overview of OpenAI crawlers, OpenAI
• Anthropic crawlers, Anthropic
• The agentic web, Cloudflare, Sept 2026
• US holiday shopping forecast, Adobe, Sept 28, 2026
• Web Bot Auth HTTP signatures draft, IETF
• Known bot-traffic exclusion, Google Analytics Help
• The verifiability gap, D3 research, Aug 2026
• Web traffic nearly doubled, and blocking AI crawlers now costs sales, D3 research, Oct 2026