My friend and co-worker John Liu from The World is Meh has just written a great blog entry about Google’s apparent indexing of webpages despite explicit prohibitions from the site’s robots.txt file. Take a look at his thoughts on the issue.

Is this just a glitch in the Google matrix or the end of the search engine’s gentlemen’s agreement with webmasters? I hope it’s the former, not the latter. I did some testing with my own robots.txt file in the Google Webmaster Tools Robots.txt Analyzer and I was assured by the analyzer that the restricted URLs in the file would not be spidered when the Googlebot visited my site. For the moment, it seems Google is crawling everything and sending pages “blocked” by robots.txt to its notorious “repeat the search with the omitted results included” section of search results.

Geez… Google mistake or not, you have to do better than this. There was a reason I asked you not to spider this file!

FREE Video Training for Internet Marketers

Get insider tips on how to make more money from your internet marketing efforts as well as how to bring thousands of new visitors to your website. Here’s what you get:

• Ultimate Entrepreneur eCourse
• The Online Profit Training
• SEO Traffic Building Coaching Call (pre-recorded)
• Blogging for Newbies eCourse

We respect your privacy. Your information will not be shared with any third party and you can unsubscribe at any time

About author:

Everett Whitehead is an experienced emarketer who specializes in search engine optimization, SEM, and other areas of online marketing. Active in online business since 2002, Everett is privileged to have worked with many well-known brands over his career. If you would like to retain a Boston SEO Consultant or would like an SEO question answered, he can be contacted by clicking here.

All entries by Everett Whitehead