Google Breaking Covenant with Webmasters? Effectiveness of Robots.txt in Doubt
Nov 1st, 2007 | By Everett Whitehead | Category: Search Engine Optimization (SEO)
My friend and co-worker John Liu from The World is Meh has just written a great blog entry about Google’s apparent indexing of webpages despite explicit prohibitions from the site’s robots.txt file. Take a look at his thoughts on the issue.
Is this just a glitch in the Google matrix or the end of the search engine’s gentlemen’s agreement with webmasters? I hope it’s the former, not the latter. I did some testing with my own robots.txt file in the Google Webmaster Tools Robots.txt Analyzer and I was assured by the analyzer that the restricted URLs in the file would not be spidered when the Googlebot visited my site. For the moment, it seems Google is crawling everything and sending pages “blocked” by robots.txt to its notorious “repeat the search with the omitted results included” section of search results.
Geez… Google mistake or not, you have to do better than this. There was a reason I asked you not to spider this file!
