Why Pages Disallowed In Robots.txt Still Appear In Google

Meaning of Robots.txt

Meaning of Robots.txt

Robots.txt is a useful file which places in your websites root and controls how search engines index your pages. One of the most useful declarations is Disallow it stops search engines accessing private or irrelevant sections or pages of your website, e.g.

Disallow: /temp/

Disallow: /mypage.html/

You can Even Block Search Engines Indexing Every Page on Your Domain, e.g.:

User-agent: *

Disallow: /

Blocked Pages can Still Appear in Google HOW?

Take a little while to understand how and why it happens. Assume you have a page at http://www.abc.com/mypage.html containing confidential information about your companys new coupon codes project. You may want to share that page with partners, but dont want the information to be public knowledge just yet. Therefore, you block the page using a declaration in http://www.abc.com/robots.txt:

User-agent: *

Disallow: /mypage.html

A few weeks later, youre searching for coupon codes in Google and you found http://www.abc.com/mypage.html at 1st Page of Google. How could this happen? It means, Google abides with your robots.txt instructions, isnt?

However, this is not a violation of robots.txt rules. This happens because of very simple reason that Google found your link from elsewhere, means http://www.abc.com/mypage.html might be linked from any external website, so Google caught you from there. Meta information also comes from that particular external link, not from your page content.

There are Several Solutions that will Stop Your Pages Appearing in Google Search Results:

Set a no index Meta Tag: Google will never show your page or follow its links if you add this code to your HTML head section:

Use the URL removal tool: Google offer a URL removal tool within their Webmaster Tools.

Add authentication: Apache, IIS, and most other web servers offer basic authentication facilities. The visitor must enter a user ID and password before the page can be viewed. This may not stop Google showing the page URL in results, but it will stop unauthorized visitors reading the content.

by: Bob Smith123

POST COMMENT

Seo Consulting California: Implementing Google Analytics For Leveraging Seo Who Are The Ones That Make Money With Google Adsense? Increasing Google Pagerank What is Google Instant and Will it Affect You? Ppc Basics Keyword Research Free Google Tools Make Money with Google | Earn Money How Google Traffic Works 7 SEO Things Google Really Hates - If in doubt Google will throw you out Do you think like a Search Spider - Google Search Bot? Integrate Tweets into Real Time Searches. Find How Google Ranks Tweets? The 3 second rule with Google Instant Google Instant…quick(er) Results ..saves time but? Want Your Company To Be Found On Google? Hire Seo Consultants