Escape google Supplementary Results jail with URL rewrites?

HB · phpBB SEO Team Joined: 16 Oct 2006 Posts: 812

I migrated my forum to the advanced rewrite mod + zero dupe a couple weeks ago. During this period, google and yahoo have furiously updated their indices to the new title-injected URLs and dropped lots of viewtopic.php?p= type dupes. Nice. Very Happy

This week I noticed what might be an unexpected benefit. Hundreds of my forum's threads were in google's Supplementary Results index. While the pages in the main index are updated frequently, those in the SR languish. With the new rewritten URLs, google has picked up the new versions and indicates they are in the main index. Very nice.

Note: To find those pages of my forum in the SR, I searched on "viewtopic". Not too surprisingly, the pages in the main index were already redirected to the remapped versions after a couple weeks, so there were few hits there. The majority of the "viewtopic" matches are from the SR; to check if they had an existing counterpart in the main index, I searched on those with a very unique title. Google returned both matches, one from the main index (rewritten URL) and another from the SR (viewtopic URL).

Technically they're dupes. Then again, I'd rather have duplicate pages between the main index and the SR index than only the SR. So what do you think, is this a legitimate way out of Supplementary Result jail?

dcz

I bet the lasting viewtopic.php?t=xx like urls will just soon disappear.

Double listing is here just showing us nothing is live with SEs, all DC must not already be aware of the update, but they will Wink

About the supplemental results, I think this is as well on of the zero duplicate benefit, it's stronger than a robots.txt disallow as bots will really not be able to browse any dupe.
Here, even if they try, they cannot trick our url standard Very Happy

This together with gym sitemaps makes indexing real easy for bots, they now what to crawl and cannot get lost out of our standard.
We for sure end up running forums requiring a quite less resources to be crawled, and this could explain why they hesitate a bit less to do it more often (eg in the main index).

A interesting question is : will phpBB SEO be responsible for Google needing more servers to handle all these new forum being indexed so well, or is it that they in the end will not need more servers (even less ?) because the zero dupe compensate the better indexing (as many forums could end up with quite less indexed urls after the duplicate reduction) Very Happy

What is for sure is we do make things easier for SEs Wink

++

HB · phpBB SEO Team Joined: 16 Oct 2006 Posts: 812

dcz

Well, I don't think we'd really want the sitemaps to be this strict.

Most of the web sites do not list all their url in sitemaps, as here for example, we still need natural crawl to be fully indexed.
Of course the best would be to list all of our web sites url in sitemaps, but, we're far from it. It's done for the forum, at least for the 50 000 first topic of each forums, but about the rest.
One could think online sitemaps service could solve this, but, most of the time, they end up being used improperly and list bad and duplicates urls so, I think regular crawling is far from dead.

But, the sitemaps do help out bot to find the most usable urls, for example mx-system.com was not indexed at all (almost) because of sid, and despite it's PR7, and mx Google sitemaps changed this drastically, Google was all the sudden able to find sid less url in sitemaps and indexed them all real fast.
Even though the web site was and is still showing sids for guests.

++