subject: About Robots Text File [print this page] Robot text file optimization is really important, if it did not search engine bots can crawl your website for confidential information and can display them in their respective search engines or if you are using the robots txt file. But it is not written properly then it could be a scenario that can block search engines from crawling the website.
We restrict our website to block spiders can crawl parts. Restricted parts of our website our website that we have to be indexed in search engines and do not want to get some unwanted visitors means those links. For example: -
How many of you would be interested in your search engine indexing administration page?
In the past when I in some cases website owners were mad at their website using the robots.txt saw were scared as they will damage the SEO. Well this is not true we can use robots txt properly, so we restricted exclusively to prevent crawler from crawling the link, and it is not restricted to every other link which we also do damage to our website SEO will have to crawl .. We accomplish this task by using robots. Txt can.
Before we discuss the optimization and can take full advantage of the robot can. Txt, we should first discuss the basic concepts of robots. Txt
A robot.txt text file that is to be placed in the root folder of your web server (where you place your website's index page). You can make this file in Notepad simple one. The various search engine bots that crawl the website may or may not be part of the index must not partition. We use it to instruct the bots crawl or we instruct them that they must crawl or index can not be certain areas of the website can be stopped by our website. We can use the same robots txt. Bots allow for different directions.
Even if you crawling or indexing your website do not protect any area, even though you should use robots. Txt as an open invitation to the search engines to crawl your entire website can act as.
There are several scenarios in which some parts of your website search engine bots from crawling may be interested may stop.
For instance
1) Protect your website's administration panel.
2) The keep pages from search engine that are under construction.
3) Protect the directory that you want to be indexed as CGI-bin
4) Protects the pages that email addresses used by spammers as they can be indexed in the search engines found.
Reasons can be various but the solution is same, that is using robots.txt now let's start writing in robots.txt
Basic syntax of robots.txt is
User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]
You can repeat these lines for blocking different directories or giving different instructions to different spiders. Let's get in few examples that will make it more clear.
Example 1) Exclude a file named private.html in private folder from being crawled by Google-bot.
Solution 1) In this scenario you can write the following code in robots.txt
User-Agent: Googlebot
Disallow: /private/private.html
Example 2) Exclude a folder named private from being crawled by search engine.
Solution 2) In this scenario you can write the following code in robots.txt
User-Agent: *
Disallow: /private/
Example 3) Instruct Search Engine bots to crawl and index everything on the website.
Solution 3) In this scenario you can write the following code in robots.txt
User-Agent: *
Disallow:
Example 4) Instruct Search Engine bots that they should not crawl or index any part of the website.
Solution 4) In this scenario you can write the following code in robots.txt
User-Agent: *
Disallow: /
Example 5) Exclude multiple folders (private1,private2,private3) from being crawled by search engines.
Solution 5) In this scenario you can write the following code in robots.txt
User-Agent: *
Disallow: /private1/
Disallow: /private2/
Disallow: /private3/
Example 6) Instruct Google-bot to crawl everything on the website and instruct Alexa bot that it should not crawl any part of the website
Solution 6) In this scenario you can write the following code in robots.txt
