subject: What Can A Robots Txt File Do For Website Seo [print this page] First of all lets get some definitions listed to better understand the usefulness of robots.txt file:
Robots: robots are software agents that visit your website and read your robots.txt file for access information then continue to read the rest of your website.
robots.txt File: this file is a file that has the commands to allow or disallow Robots to access specific or all directories of your website.
Crawling: is the process of reading your website directories and pages by search engine robots.
When you make a new website, usually you need to have a robots.txt file to dictate how robots or spiders will crawl your website. If you do not have a robots.txt file then search engines will crawl all pages and directories on your website.
Imagine your website being an apple basket and each component of your website (pages, posts, media...) are apples. If you have apples with very good Quality and others with relatively low Quality, then anyone looking at this basket will also see the low quality apples and most probably wont buy the basket.
Like the apples basket your website has good quality pages and bad quality pages. Bad quality pages include, login pages, tags pages, category pages, and possibly random folders and files uploaded on your website that you don't want search engines to crawl or see.
If search engines see those low quality pages like login pages or duplicate content pages (like tags and categories) it will hurt your website SEO. Duplicate content resulting from tags and categories might get those pages to outrank your static pages and posts. Every link to your tags and category pages is technically wasted since it does not link to a static or post page.
Using the robots.txt file, you can tell search engines what directories and pages you don't want them to look at. Search engines will only see what you want them to see resulting in an overall better website content and quality.
Restrict Private Pages and Content through robots.txt
You can restrict directories and pages that are private. Private pages or files can be kept out of search engines in the following manner.
User-agent: *
Disallow: /MyFiles
Disallow: /PrivateD
Disallow:/Pages/Login.php
In the above example we are allowing all User-agents by typing a star (*) after it. The second line will not allow search engines to crawl anything in the directory http:// www. yourwebsite.com/MyFiles. The same goes for PrivateD. The third line means that you are allowing all pages except for Login Page.
You can now play with your robots.txt file and make sure you show search engines the best of your website. Thanks for reading.
