AI glossary: every agent and term, in plain words
This glossary covers the AI agents you'll see in your logs, plus the terms, acronyms, standards, and vendors around them. Each entry says what it is in one or two plain sentences.
Last updated November 7, 2026 · 86 entries · 22 AI agents
#
A robots.txt name that OpenAI, Google and Microsoft plan to honor as an AI training opt-out. It doesn't stop crawling, and a User-agent: * group never counts.
A
An open standard, started by Google, that lets AI agents from different companies talk to each other and hand off tasks.
OpenAI's standard, built with Stripe, for selling through ChatGPT. Stores start by sending a product feed, then can let customers buy in the chat.
Work to get your pages cited in AI answers from tools like ChatGPT, Perplexity, and Google AI Overviews. Also called GEO.
Software that visits your site or uses a service on someone else's behalf. See AI agent.
A web browser with an AI agent built in that can click, fill forms, and buy for its user. Examples: Comet, ChatGPT agent, Claude in Chrome.
Shopping where an AI agent finds, compares, or buys products for a person.
Software that reads or acts on websites for an AI company or for a person. The four common kinds are training crawlers, search crawlers, assistants, and shopping agents.
A GA4 channel, added in May 2026, for people who reach your site from AI apps such as ChatGPT, Gemini and Copilot. It counts people who click through, not the agents themselves.
Google's AI-written summaries at the top of search results, built from pages Googlebot crawls.
A visit from a person who clicked a link inside an AI answer. Shows in analytics as a referral from sites like chatgpt.com.
A CDN and security company. Its bot tools detect and stop malicious bots.
Amazon's crawler for its products and services, including Alexa answers.
A tool that replies with an answer instead of a list of links. ChatGPT search, Perplexity, and Google AI Overviews are answer engines.
A Google-led standard that proves a person approved an agent's purchase before it pays.
Apple's crawler for Siri and Spotlight search results.
A robots.txt name that controls whether Apple uses your pages to train its AI models. Applebot still crawls for search.
An agent that reads a page while a person waits for an answer, like ChatGPT-User or Claude-User. Also called a user-triggered fetcher.
B
Microsoft's crawler for Bing, which also supplies results to Microsoft Copilot.
Any automated software making requests to your site. AI agents are one kind of bot.
Tools that detect and control bots, usually to stop attacks, fraud, and scraping.
ByteDance's crawler, widely reported to collect data for AI training.
C
A test meant to stop bots, like picking images or ticking a box. It also stops the agents shopping for your customers.
Common Crawl's crawler. Its free public archive feeds many AI training sets.
A network of servers that delivers your site quickly worldwide and often filters traffic at the edge. Cloudflare, Akamai, and Fastly are CDNs.
OpenAI's agent that browses, compares, and completes tasks like purchases for a ChatGPT user.
OpenAI's assistant agent. Reads a page when a ChatGPT user asks about it. OpenAI says robots.txt rules may not apply to it, because a person started the request.
Anthropic's browsing agent, which works in Chrome for a Claude user.
Anthropic's search crawler. Indexes pages to improve Claude's search results.
Anthropic's assistant agent. Reads a page when a Claude user asks about it, and follows robots.txt.
Anthropic's training crawler. Reads public pages to train Claude models.
A CDN and security company. Offers AI crawler controls and pay per crawl, and publishes AI traffic research on Cloudflare Radar.
Perplexity's agentic browser, which can browse and act for its user.
A nonprofit that crawls the web and publishes a free archive. Many AI models train on it.
A paid agreement that lets an AI company use your content. Most deals today are between large publishers and AI companies.
A robots.txt line asking a bot to wait between requests. Not part of the official standard, but some bots, including Anthropic's, honor it.
How many pages an AI company reads for each visitor it sends back to you. A high ratio means it takes more than it gives.
A bot that visits pages automatically to collect them. Training crawlers and search crawlers are both crawlers.
D
A bot and fraud protection company.
D3's tool that checks every AI agent that visits your site, then lets it in, charges it for the page, or stops it.
F
A CDN and edge cloud company with bot controls.
G
Google's web analytics. It counts visits that run its tag and removes known bots automatically, so most AI agent visits never show up in its reports.
Another name for AEO.
A robots.txt name that controls whether Google uses your pages to train Gemini. Googlebot still crawls for Search.
Google's crawler for Search, including AI Overviews.
OpenAI's training crawler. Blocking it keeps your pages out of OpenAI model training without affecting ChatGPT search.
H
A browser run by software with no screen. Agents and scrapers both use them, so they can look like human visitors.
A web status code meaning "Payment Required." Unused for decades, now the basis of x402 and pay per crawl.
A bot and fraud defense company that publishes research on agent traffic.
I
The industry list of known bots and spiders. GA4 uses it to filter bots out of reports.
The group that sets internet standards. robots.txt (RFC 9309) and Web Bot Auth go through it.
Checking a request's address against the list an AI company publishes for its agents. OpenAI publishes lists for GPTBot, OAI-SearchBot, and ChatGPT-User.
L
The AI model behind tools like ChatGPT and Claude. Training crawlers collect pages to build them.
A proposed markdown file at /llms.txt that points AI tools to your key pages. Evidence that AI crawlers use it is thin so far.
M
Mastercard's program for agent payments, using tokens tied to a cardholder's approval.
An open standard, started by Anthropic, for connecting AI tools to data and apps.
Meta's crawler for training AI models and improving its products.
A standard for machines to pay for services, often per request. One of the four protocols in most agent payment comparisons.
A personal AI agent that shops and gets things done for its user.
O
OpenAI's agent that checks landing pages submitted as ChatGPT ads. OpenAI says its data is not used for training.
OpenAI's search crawler. Sites that block it won't appear in ChatGPT search answers.
P
A Cloudflare feature that lets a site charge AI crawlers for each page they read, using HTTP 402.
Perplexity's assistant agent. Reads a page when a Perplexity user asks about it.
Perplexity's search crawler. Indexes pages for Perplexity answers.
An AEO platform that tracks how brands appear in AI answers and publishes research on AI search.
Q
When an AI tool turns one question into several searches behind the scenes. Pages that answer related questions get found more often.
R
Capping how many requests a visitor can make in a given time. Slows heavy crawlers without blocking them.
The official robots.txt standard, published by the IETF in 2022. It says robots.txt rules don't control access.
A file at the root of your site that tells bots which pages they may crawl. Each bot decides whether to follow it.
An open standard for publishers to state AI licensing terms in a file machines can read.
S
A bot that copies content or prices, often without permission. Scrapers often use famous agent names to get past your rules.
An agent that indexes pages so they can appear in AI answers, like OAI-SearchBot or Claude-SearchBot.
An agent that browses, compares, and buys for a person, like ChatGPT agent, Comet, or Muse.
An agent that signs each request with a key, so your site can prove who sent it. See Web Bot Auth.
Faking a famous agent's name, like GPTBot, to get past a site's rules.
T
Visa's standard to help stores recognize agents shopping for a real cardholder.
A company that helps publishers charge AI bots for access to content.
An agent that reads pages to train AI models, like GPTBot or ClaudeBot. Rarely sends visitors back.
U
A Google-led open standard for agent shopping and checkout.
The name a visitor sends with every request, like "GPTBot/1.4." Easy to fake, so treat it as a claim.
V
A bot whose identity has been checked by IP list or signature, not just by name.
W
A filter in front of your site that blocks attacks. Often where bot rules live.
A draft IETF standard that lets an agent sign each request, so your site can verify the sender.
X
A payment standard built on HTTP 402, so an agent can pay per request, typically in stablecoins. Now run by the x402 Foundation at the Linux Foundation.
Z
A search where the person gets the answer without clicking through to any site. AI answers make it more common.
A company publishing guides and tools on how sites get found and paid by AI agents.
Check your site
Which of these agents does your site let in?
Check your robots.txt