What Can A Robots Txt File Do For Website Seo

First of all lets get some definitions listed to better understand the usefulness of robots.txt file:

Robots: robots are software agents that visit your website and read your robots.txt file for access information then continue to read the rest of your website.

robots.txt File: this file is a file that has the commands to allow or disallow Robots to access specific or all directories of your website.

Crawling: is the process of reading your website directories and pages by search engine robots.

When you make a new website, usually you need to have a robots.txt file to dictate how robots or spiders will crawl your website. If you do not have a robots.txt file then search engines will crawl all pages and directories on your website.

Imagine your website being an apple basket and each component of your website (pages, posts, media...) are apples. If you have apples with very good Quality and others with relatively low Quality, then anyone looking at this basket will also see the low quality apples and most probably wont buy the basket.

Like the apples basket your website has good quality pages and bad quality pages. Bad quality pages include, login pages, tags pages, category pages, and possibly random folders and files uploaded on your website that you don't want search engines to crawl or see.

If search engines see those low quality pages like login pages or duplicate content pages (like tags and categories) it will hurt your website SEO. Duplicate content resulting from tags and categories might get those pages to outrank your static pages and posts. Every link to your tags and category pages is technically wasted since it does not link to a static or post page.

Using the robots.txt file, you can tell search engines what directories and pages you don't want them to look at. Search engines will only see what you want them to see resulting in an overall better website content and quality.

Restrict Private Pages and Content through robots.txt

You can restrict directories and pages that are private. Private pages or files can be kept out of search engines in the following manner.

User-agent: *

Disallow: /MyFiles

Disallow: /PrivateD

Disallow:/Pages/Login.php

In the above example we are allowing all User-agents by typing a star (*) after it. The second line will not allow search engines to crawl anything in the directory http:// www. yourwebsite.com/MyFiles. The same goes for PrivateD. The third line means that you are allowing all pages except for Login Page.

You can now play with your robots.txt file and make sure you show search engines the best of your website. Thanks for reading.

by: www.Web-Design-SEO-Services.com

POST COMMENT

How To Take Advantage Of A Free Ipad Promotion 4 Multiple Streams Of Residual Income Promotion Tips How To Ace Fantastic Deals For Promotional Glass Wares Why Has A Website? Pocket Watches and Breitling Replica Watches: Classy, Stylish Promotional Items The Best Web Site Hosting Service - What Are The Features Manually submit a website through an SEO service firm Travel Websites Good Source Of Travel Information 10 Steps to Website Globalization Professional Website Design - What To Look For To Make The Best Impression Huge Rush in Website Optimization That Improves Sales Target – Think Tank InfoTech India How to choose Promotional Gifts that will get noticed Website Development - Basic Knowledge And Guidelines To Choose A Website Development Company