Robots Txt for PHPBB mod rewrite

[sceltic]

What can be improved or needs to be over the old able2know robots txt suggested standard.

Here is that robots txt

Code: Select all

OPEN ]------------------------------------------

robots.txt

Disallow: /phpbb/post-*.html$
Disallow: /phpbb/updates-topic.html*$
Disallow: /phpbb/stop-updates-topic.html*$
Disallow: /phpbb/ptopic*.html$
Disallow: /phpbb/ntopic*.html$


I believe some of the above were done prior to wild cards the * part being realised as no use.

For instance now that would be more correct

as
Disallow: /phpbb/post-

[dcz]

I'd say you go for :

Code: Select all

Disallow: /phpbb/post-
Disallow: /phpbb/updates-topic.html
Disallow: /phpbb/stop-updates-topic.html
Disallow: /phpbb/ptopic
Disallow: /phpbb/ntopic
Disallow: /phpbb/index.php?
Disallow: /phpbb/posting.php
Disallow: /phpbb/modcp.php
Disallow: /phpbb/login.php
Disallow: /phpbb/privmsg.php
Disallow: /phpbb/viewonline.php


And you could even add those too :

Code: Select all

Disallow: /phpbb/faq.php
Disallow: /phpbb/profile.php
Disallow: /phpbb/groupcp.php


But it's as you want, it's just in case you don't want spammer to love your memberlist too much, groucp and faq are mostly interesting to members, not really a search target.

I'll soon start posting article on how to analyse a site indexing and such, to find out better what should be done.
++

[sceltic]

Good information.