URLs restricted by robots.txt

[Silverado05]

Ok, I was going through my google sitemaps account checkout the latest data. While I have over 11,000 pages indexed and they seem to be vauable content It notfied me that I have (1107) URLs restricted by robots.txt. Now I was scanning the URLs it restricted and some are legit like member profiles, etc. But I also noticed it is restricting the URL's of acutally posts I.E.

http://www.texascampingforum.com/forum/post358.html

So wouldn't that need to be unresctriced as that is vaulable content? Thier is several 100 URL's like that, that go to post.

Here is the robot text:

[dcz]

Working on a new design, nice

So everything seems normal here.

We have to disallow post as they are the messages URLs (viewtopic.php?p=), the number one duplicate source in a phpBB forum.

Same for natural URLs vewtopic.php ...


Then, the message about accessing your home page is nothing. Maybe the bot could not load the page once (network failure or server reboot) and the stats are not updated yet (it's far from being live).
Because your home page is cached.

I would not worry about any of those, especially your robots.txt. Allowing post URLs can make one think it's faster for indexing, and it is some how true, at least without mx Google sitemaps, as the message URL do show up on the forum index (one page before the topic's in depth).

But, this is not a good thing to do, because all those are going to be full or partial duplicates of the topic's URLs (at least same title).
The result would be a lot more cached page, but a lot fewer PageRank (because of the content dilution) and thus less good results in search engines results.
One URL, one page, one title, the simple the better (bots are not as smart as we are, we'd better not confuse them).

++

[Silverado05]

So to sum it up then it would be best to leave things as it is and keep Disallow: /forum/post and/or Disallow: /forum/viewtopic.php those on the robot.txt?

Also another interesting thing I saw was Googlebot had successfully accessed my home page on Dec 31, 1969? 1969??? LoL Google wasn't even around then, much less the internet.

[dcz]

1969?
You know what, I always new Google was a rock star


This reminds us Google sitemaps is still beta, it works nice, but some things like this can happen.

++

[lavinya]

vavvv

1969