subject: SEO – What is Robots.txt? [print this page] SEO What is Robots.txt? SEO What is Robots.txt?
SEO is an important part of online business and always try to make their website search engine friendly. When the search engine crawls your site they will check robot.txt file and seeing either they have the permission for entering your site or not. So, according to SEO kind of view, it is important to make your site search engine friendly.
Robot exclusion standard is also known as robots exclusion protocol or robot.txt protocol. Robot.txt is a non-html text file which we put on our site to show search robots which page we would like them not to visit. This is so important that robot.txt is not 100% reliable coding to prevent search robots. It is not firewall or a kind of password protection. It is something like making a note as "Please, do not enter" on an unlocked door - e.g. you cannot prevent thieves to coming in, but good person will not open the door.
According to SEO kind of view the location of robot file is very important; it must be in main directory otherwise user agent means search engine will be not able to find it because they do not search the whole site for a file named robot.txt. If you have sensitive data on your web site that you do not want the world to see, you will also prefer that search engine does not index these pages (by doing it offline on a separate machine) and other way is that to tell spider to keep away from these page. In addition, it is techniques which help us to tell search engine that not to enter that specified pages which you do not want to index. There are some search engines that do not follow the rules. So, in that case what do you do with these robots that do not obey the robot.txt rules? You should set up a .htaccess file to instruct the bad search engine to stay away.
