Using robots.txt to block parts of the forum

[MM84]

Hi people, I hope you can help me!

I've recently upgraded to phpbb3 and so far only installed one mod- google sitemap from phpbbhacks.com.

It's working brilliantly, except that google keeps crawling the hidden moderators forum and as a result, you can read private threads in the google index.

How can I stop google from indexing it?

The forum is:

viewforum.php?f=9

Please help! It's a matter of urgency!!

Thanks,

MM

[dcz]

Are the private forums really cached in Google's index, or do you only see the login form in there ?

phpBB3 can allow Bots to crawl private forums, but you need to set the authorisation to do so. It's not the default behaviour, so the first thing to make sure is : are the private forums and topic inside really cached.

And, make sure as well that the sitemap you are using does not provide bots with unnecessary links to pages they cannot load.

++

[mhmdkhamis]

i noticed google dont into in any forums in my forum but my topics get cash

what do you think dcz

[dcz]

Do you mean that you do not see the Google bot but it's still caching public content ?

Well, you must have missed the bot

++

[emielvenlo]

I have had this problem to after upgrading. This is because the permissions of the group "bots" have to be set. You have to go to group permissions of the bots and alter them so they can only see the public forums.

[mhmdkhamis]

Do you mean that you do not see the Google bot but it's still caching public content ?

Well, you must have missed the bot

++

yes dcz i see google bot in my forum index page only but my topics get cash good

[dcz]

No forum cached ?

[mhmdkhamis]

No forum cached ?

and my forums get cash good but no google bot can see into my forums

[dcz]

Then I would not really bother.
The chances are actually few to see a bot on a given page, you could need to wait for like two or three days to see the Google bot on a given forum.
And it would not be bad to have Google checking the forums every couple days, that's actually a nice rate.

++

[MM84]

I have had this problem to after upgrading. This is because the permissions of the group "bots" have to be set. You have to go to group permissions of the bots and alter them so they can only see the public forums.

Thanks!