Robots.txt

Robots.txt files play an important part of the WordPress setup. It is used to inform search engine spiders of the location it can crawl and the places you do not want it to crawl.