TermAI bots & crawlers

Crawl-delay

A robots.txt line asking a bot to wait between requests. Not part of the official standard, but some bots, including Anthropic's, honor it.

Source: Anthropic

Last updated November 7, 2026

← All entries

See which AI agents visit your site, and what they came for.

Sign up free

DNA is free for every site.