I migrated my forum to the advanced rewrite mod + zero dupe a couple weeks ago. During this period, google and yahoo have furiously updated their indices to the new title-injected URLs and dropped lots of viewtopic.php?p= type dupes. Nice.

This week I noticed what might be an unexpected benefit. Hundreds of my forum's threads were in google's Supplementary Results index. While the pages in the main index are updated frequently, those in the SR languish. With the new rewritten URLs, google has picked up the new versions and indicates they are in the main index. Very nice.

Note: To find those pages of my forum in the SR, I searched on "viewtopic". Not too surprisingly, the pages in the main index were already redirected to the remapped versions after a couple weeks, so there were few hits there. The majority of the "viewtopic" matches are from the SR; to check if they had an existing counterpart in the main index, I searched on those with a very unique title. Google returned both matches, one from the main index (rewritten URL) and another from the SR (viewtopic URL).

Technically they're dupes. Then again, I'd rather have duplicate pages between the main index and the SR index than only the SR. So what do you think, is this a legitimate way out of Supplementary Result jail?

I bet the lasting viewtopic.php?t=xx like urls will just soon disappear.

Double listing is here just showing us nothing is live with SEs, all DC must not already be aware of the update, but they will

About the supplemental results, I think this is as well on of the zero duplicate benefit, it's stronger than a robots.txt disallow as bots will really not be able to browse any dupe.
Here, even if they try, they cannot trick our url standard
This together with gym sitemaps makes indexing real easy for bots, they now what to crawl and cannot get lost out of our standard.
We for sure end up running forums requiring a quite less resources to be crawled, and this could explain why they hesitate a bit less to do it more often (eg in the main index).

A interesting question is : will phpBB SEO be responsible for Google needing more servers to handle all these new forum being indexed so well, or is it that they in the end will not need more servers (even less ?) because the zero dupe compensate the better indexing (as many forums could end up with quite less indexed urls after the duplicate reduction)

What is for sure is we do make things easier for SEs

++

dcz wrote:This together with gym sitemaps makes indexing real easy for bots, they now what to crawl and cannot get lost out of our standard...

A interesting question is : will phpBB SEO be responsible for Google needing more servers to handle all these new forum being indexed so well, or is it that they in the end will not need more servers (even less ?)...

I believe at some point sitemaps will surplant robots.txt rules. Correctly implemented, they tell search engines precisely what URLs are legitimate content. Google webmaster tools, for example, could easily show a webmaster the URLs in its index that are not in the sitemap and confirm they should be removed. This sort of "hint" would clean duplicates and deadends faster than the zero dupe.

More accurate search engine indices, more efficient and more timely updates. What's not to like?

Well, I don't think we'd really want the sitemaps to be this strict.

Most of the web sites do not list all their url in sitemaps, as here for example, we still need natural crawl to be fully indexed.
Of course the best would be to list all of our web sites url in sitemaps, but, we're far from it. It's done for the forum, at least for the 50 000 first topic of each forums, but about the rest.
One could think online sitemaps service could solve this, but, most of the time, they end up being used improperly and list bad and duplicates urls so, I think regular crawling is far from dead.

But, the sitemaps do help out bot to find the most usable urls, for example mx-system.com was not indexed at all (almost) because of sid, and despite it's PR7, and mx Google sitemaps changed this drastically, Google was all the sudden able to find sid less url in sitemaps and indexed them all real fast.
Even though the web site was and is still showing sids for guests.

++