Robots.txt

Robots.txt is a file that gives instructions to search engine crawlers about which pages or sections of a website they can or cannot access.