Hi all, hope you can help. Google is still looking at areas of my forum that I do not want it to waste it's time on, for example it still views private messages and profile

Below is my robots txt file. Any advice appreciated.

Thanks

User-agent: *
Disallow: /forum/private/
Disallow: /forum/admin/
Disallow: /forum/db/
Disallow: /forum/docs/
Disallow: /forum/cache/
Disallow: /forum/cgi-bin/
Disallow: /forum/files/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/language/
Disallow: /forum/tmp/
Disallow: /forum/templates/
Disallow: /forum/common.php
Disallow: /forum/memberlist.php
Disallow: /forum/memberlist.php?
Disallow: /forum/config.php
Disallow: /forum/faq.php
Disallow: /forum/groupcp.php
Disallow: /forum/login.php
Disallow: /forum/modcp.php
Disallow: /forum/profile.php?mode=viewprofile&u=
Disallow: /forum/profile.php?mode=editprofile
Disallow: /forum/profile.php?mode=email&u=
Disallow: /forum/printview.php
Disallow: /forum/privmsg.php
Disallow: /forum/privmsg.php?
Disallow: /forum/ranks.php
Disallow: /forum/search.php
Disallow: /forum/viewonline.php
Disallow: /forum/profile.php
Disallow: /forum/profile.php?
Disallow: /forum/posting.php?
Disallow: /forum/posting.php

you dont have to disallow all
just put there this

Code: Select all

User-agent: *
Disallow: /forum/viewtopic.php?
Disallow: /forum/viewforum.php?
Disallow: /forum/index.php?
Disallow: /forum/posting.php?
Disallow: /forum/groupcp.php
Disallow: /forum/profile.php?
Disallow: /forum/memberlist.php
Disallow: /forum/search.php?
Disallow: /forum/login.php
Disallow: /forum/faq.php

Thanks for the reply, will the viewtopic, search or viewforum ban stop google from indexing posts though?

if you are not url rewriting yes, this robots.txt will prevent all the natural link indexing in phpBB forums.

Thanks guys can anyone help me write a simple robots.txt that simply stops google looking at private messages and viewing profile etc? I want it to index all my posts.

Thanks

i thins Pm can bot see in default and how can he does it, he can post anything, and profile is /forum/profile.php

Jimcanswim wrote:Thanks guys can anyone help me write a simple robots.txt that simply stops google looking at private messages and viewing profile etc? I want it to index all my posts.

Thanks

Code: Select all

User-agent: *
Disallow: /phpbb/posting.php
Disallow: /phpbb/groupcp.php
Disallow: /phpbb/profile.php
Disallow: /phpbb/memberlist.php
Disallow: /phpbb/search.php?
Disallow: /phpbb/login.php
Disallow: /phpbb/faq.php

will disallow profile, memberlist, group pages, search result pages, login page and faq.

Replace "phpbb/" with your actual phpBB path if any, or with nothing in case phpBB is installed in the domain's root.


++

Thanks dcz, I'll give it a go.

DCZ, I implemented your change but the bot is still looking at profile etc. Whats going on?

Any Ideas?

Cheers

and how you know it you see any new indexed profiles?
o dont think so you see only in who is online

Jimcanswim wrote:DCZ, I implemented your change but the bot is still looking at profile etc. Whats going on?

Any Ideas?

Cheers

Well, a new disallow in the robots.txt could need years to be fully taken into account by bots. Google should be faster than other, but it depends, it can take a long time especially if the pages previously weren't disallowed and already indexed.

They should though not be cached again, new profiles should not be cached as well.

++

Thanks for the replies guys. When I go to my admin area I see the google bot ip address still looking at profile etc. That's how I know it is still looking at the files, despite the robots.txt file.

Cheers

dont trust that who is online and where is it.Wait some time and you will see the results

dcz wrote:Well, a new disallow in the robots.txt could need years to be fully taken into account by bots. Google should be faster than other, but it depends, it can take a long time especially if the pages previously weren't disallowed and already indexed.

If bots traversing old profile links really bothers you, remap them to something else and redirect the "old" URLs to your forum's index. I've noticed that search engines take an ETERNITY to forget about a URL once discovered, even if you return "410", but they'll take into account 301 "permanently moved" return code in a couple weeks.

I've added an auth check to memberlist.php, groupcp.php and usercp_viewprofile.php on my forum so that these pages can only be viewed by users who are registered and logged in. I did this mainly because I don't want humans looking at those pages unless they're members, but partly also because I too had noticed that Google seems very slow to adjust its behaviour to correspond to new modifications to robots.txt. It might be a bit of a sledgehammer solution but you could do the same thing only with an IP check set to exclude googlebot's IPs instead of an auth check.