Google's Supplemental Results
Posted by Miles Evans
As I was running through my daily web chores today I noticed something peculiar on Big Daddy server: http://64.233.167.104. I did a site: command and instead of the usual 100,000 or so results I saw the dreaded supplemental results notification. I had lost 98,000 indexed pages and left with a paltry 108 pages! The website in question is an old and trusted client and the website is primarily a large Vbulletin based forum. As of yet the changes have not reflected on non Big Daddy servers, but that sinking feeling in my gut tells me that this will in fact happen in the next 48 hours or so.
The official Google speak on supplemental results:
Supplemental sites are part of Google’s auxiliary index. We’re able to place fewer restraints on sites that we crawl for this supplemental index than we do on sites that are crawled for our main index. For example, the number of parameters in a URL might exclude a site from being crawled for inclusion in our main index; however, it could still be crawled and added to our supplemental index. The index in which a site is included is completely automated; there’s no way for you to select or change the index in which your site appears. Please be assured that the index in which a site is included does not affect its PageRank.
Huh? The reason pages go into the supplemental results is they are not being crawled – not that they contain crappy content. If the page at some point gets crawled again it is no longer supplemental. You can see why large discussion forums are being affected, and don’t think those of us with 250,000 indexed pages were not nervously waiting for this tree to fall. Now this can likely be overcome with some better SEO on these boards, and I for one did not really focus on much more than the basics for our Vbulletin board, but one has to wonder if this even recoverable. Many webmasters are reporting losses of all pages but their home page. Ouch!
Another added bonus of getting the supps is your pages tend to have an old description in their cache (since they are not crawled). You will often see them with a very old date. In other words, once your site is sent to Google Hell it is not going anywhere unless you make some changes.
Ok so why is this happening?
One very real possibility is the new changes are due to a temporary cache shift, a blooper, or some other dev related issue. In that case Google engineers are likely drinking a lot of coffee right now as many large and respected websites have lost 90% of their pages from the index.
Another possibility is this update has to do with duplicate content penalties.
Still another possibility is it has to do with keeping a consistent navigation through your website as Matt Cutt’s recently dropped at the SES in NY.
Specualtion aside I don't work for Google so basically I have no freaking idea what's up with G today. All I know is it sucks.
A quick scan of the usual blogs and forums didn’t reveal much but I eventually found some discussion around the recent supplemental results update at webmasterworld. Nothing yet today from Matt Cutt’s or GoogleGuy and I don’t expect there to be. Yep it’s Valium time.
Tell me this stuff isn’t heart attack food...and with a hang over no less. I would love to hear details and opinions from others who caught the supps today if you're out there.
Posted Mar 03, 2006 at 10:16 AM | Permalink | Trackback URL | Del.icio.us | DIGG!
Comments
As an update. Googleguy offered the following in the thread I quoted above at WMW:
"Based on the specifics everyone has sent (thank you, by the way), I’m pretty sure what the issue is. I’ll check with the crawl/indexing team to be sure though. Folks don’t need to send any more emails unless they really want to. It may take a week or so to sort this out and be sure, but I do expect these pages to come back to the main index."
Posted by Miles Evans on March 6, 2006 6:35 AM
I've been following the thread in WMW like a freakin hawk. It's ridiculous, to say the least. For what it's worth, this is not an issue that has affected only owners of massive forums. Owners of small & large e-commerce sites as well as owners of small yet powerful informational websites (yours truly) have been tossed into the Supplemental hell thus.
On the plus side, this will not get me to spell supplemental with an "i" (supplimental? can any of these webmasters spell?)
There have been a number of other blogs that have mentioned it, and I'm about to post something that puts them all into one place (as though it would help).
After multiple emails exchanged with the Google User Support Group, I still have no idea as to what's going to happen. There have been cases reported of even *expired domains* showing up higher than some established authority sites. Sheesh.
Posted by abhilash on March 10, 2006 12:10 PM