Crawl-delay
A robots.txt line asking a bot to wait between requests. Not part of the official standard, but some bots, including Anthropic's, honor it.
A robots.txt line asking a bot to wait between requests. Not part of the official standard, but some bots, including Anthropic's, honor it.