The verifiability gapMost known bots can’t prove who they are
August 2026 · D3 bot registry, 2,175 known bots
A third of the 344 top retail homepages Sitedex probed blocked AI agents their own robots.txt welcomes. We measured what a site in that position can actually check. Across 2,175 known bots, the answer is almost nothing.
95.8%
of known bots publish nothing a site can verify: no signing keys, no IP ranges. Their identity is a User-Agent string.
2,083 of 2,175
3.6%
advertise a Web Bot Auth key directory: a signature any site can check on the first request, from any IP.
79 of 2,175
0.6%
belong to an operator that publishes machine-readable IP ranges: Google, Microsoft, DuckDuckGo, Apple, OpenAI, Perplexity. That list is what an IP allowlist enforces.
13 of 2,175 · 6 operators
74%
of known AI assistants advertise signing keys. They’re the newest class of agent and the easiest to verify.
28 of 38
Verification by bot class
The newest agents are the most verifiable.
Share of each class advertising a Web Bot Auth key directory. Adoption runs newest-first: the assistant wave ships keys, and the crawler classes that have run for years publish nothing to check.
D3 bot registry, August 2026 snapshot. Blocking rates: Sitedex, 344 e-commerce homepages, August 1–2, 2026.
74%
AI assistants: browser agents and answer-time fetchers, the cohort Sitedex’s probes imitated.
n = 38
27%
AI search crawlers: they index for answer engines that cite and refer.
n = 11
8%
AI training crawlers: bulk readers building training corpora.
n = 104
0%
SEO crawlers, the largest class in the registry. None advertise keys.
n = 545
What fixes this
Sign requests with your domain.
The agent signs each request. Its public keys live at a well-known URL on its operator’s domain, and the key directory is itself signed, so serving it proves control of the keys rather than possession of a JSON file. That is Web Bot Auth (RFC 9421), and a site can verify it on the first request, from any IP.
Newer standards build on the same domain proof. ANS gives an agent a name under its operator’s domain that resolves to certificates and a tamper-evident lifecycle log: what the agent is and how it has behaved, on top of who signed the request.
Three quarters of known AI assistants already publish keys. Verification hasn’t caught up: sites still match IP allowlists, and an allowlist today covers six operators.
GET /product/42 HTTP/2
user-agent: ExampleAgent/1.0
signature-agent: "https://agent.example.com"
signature-input: sig=("@authority" "signature-agent")
;created=1754092800;keyid="poqkLGiym…"
signature: sig=:jdq0SqOwHdyHr9+r5jw3iY…:
# keys fetched once from the claimed domain:
# agent.example.com/.well-known/
# http-message-signatures-directoryThis is the check we run on live traffic.
D3 Edge verifies signatures and operator ranges on every request. Every agent on your site shows up with a checked identity, and a borrowed name gets caught.
Data: the D3 bot registry, August 2026 snapshot of 2,175 known bots merged from Cloudflare Radar’s bot directory and the open crawler-user-agents corpus, with our curation. Verification counts what operators advertise (a Web Bot Auth key directory, or a machine-readable IP-range file), not what validates; a follow-up study fetches each directory and checks. Denominators: 79 of 2,175 advertise key directories; 13 (9 agent families across 6 operators) publish IP ranges; none do both; class shares carry their n on the card. Blocking figures: Sitedex’s three-reader study (August 1–2, 2026), 344 leading e-commerce homepages, robots.txt permitting in every case. Analysis and conclusions here are our own.