Lost thousands of cached pages

arch stanton · PR1 Joined: 04 Oct 2006 Posts: 116

Something very odd has happened to the Google caching of my site.

Up until a couple of days ago, it had a couple of hundred cached pages and a few hundred more supplementals.

Now suddenly it has lost most of those pages and, after the first page of results, they are all non-cached pages and they're all forum/member.html and forum/post.html

I have no idea how this has happened, I don't recall the robots.txt page being moved or altered, so how could the member and post pages suddenly get indexed?

Also, a number of pages are old versions from December, but until recently much more modern versions (with custom page titles implemented) had appeared on the cache.

What has gone wrong and how do I put it right? Shocked

dcz

Well, could be several causes for this.

The disallow on the post urls suddenly showing up could only be because you did not uploaded the robots.txt as soon as you installed the mod rewrite and Google started to index these before it felled on the robots.txt and started to acknowledge the rules in it.
It could as well just be a Google problem, as sometime it can start showing up older urls in serps for some reasons (usually it does not last, and in all case it's not often, but I saw this couple times).
The cache's age (December) seems to tell us Google did have a small problem with it's index. I even saw him couple times starting to list again 8 month old cached pages Rolling Eyes

Anyway, things are going quite well for your forum : 1780 url listed, and all the cached ones are with the new url standard.
I only saw post and members urls after the third page, these should soon disappear. What is for sure is things are going into the right direction Wink

++

arch stanton · PR1 Joined: 04 Oct 2006 Posts: 116

Thanks dcz, helpful as ever. Smile

I think I know what the problem might be now. I remember someone being advised in another thread to let zero dupe take care of things rather than robots.txt and I think I misunderstood it and took the disallow /forum/post and forum/member entries out of robots.txt for a few days. Embarassed

Then I changed my mind and put them back in.

So what's the best thing to do now? Just do nothing or take the disallowforum/post and forum/member out of robots.txt again?

dcz

So everything gets explained Wink

And you're welcome Very Happy

For the member's profile, I do always prevent their indexing, it's up to every webmaster to decide so, but SEO wise, profile indexing is time loss for real content indexing.
I think that even if you'd like to see your user's profiles to be indexed, you should allow them once your forum starts to be fully indexed, just to make sure what's really important will be indexed faster.

Profile indexing on a big forum newly SEOed can slow down the indexing process, as there are many profile links among the forum.

About posts url, you should always keep them disallowed, and don't worry, things are going well for your forum indexing Wink

arch stanton · PR1 Joined: 04 Oct 2006 Posts: 116

HB · phpBB SEO Team Joined: 16 Oct 2006 Posts: 809

Another option is putting rel=nofollow on the outgoing postnnnn.html links in the viewtopics_body.tpl / viewforum_body.tpl templates and leaving your robots.txt alone. The zero dupe will redirect incoming postnnnn.html references to their corresponding topic and the rel=nofollow will prevent new ones from being (re)indexed.

arch stanton · PR1 Joined: 04 Oct 2006 Posts: 116

dcz

It's correct, a few days was enough for all the post and member links to be crawled, this show you how much Google want to index your board now Wink

++