robots.txt

A file at the root of your site that tells bots which pages they may crawl. Each bot decides whether to follow it.