Robots.txt

A file at a website root that provides crawler instructions about which URL paths automated crawlers may or may not request.