According to SEO kind of view the location of robot file is very important; it must
be in main directory otherwise user agent means search engine will be not able to find it because they do not search the whole site for a file named robot.txt. If you have sensitive data on your web site that you do not want the world to see, you will also prefer that search engine does not index these pages (by doing it offline on a separate machine) and other way is that to tell spider to keep away from these page. In addition, it is techniques which help us to tell search engine that not to enter that specified pages which you do not want to index. There are some search engines that do not follow the rules. So, in that case what do you do with these robots that do not obey the robot.txt rules? You should set up a .htaccess file to instruct the bad search engine to stay away.