News of roBots

How to block all other bots except the major bots as said in the title while blocking them from the basic files
Disallow: /viewtopic.php?
Disallow: /viewforum.php?
Disallow: /groupcp.php
Disallow: /profile.php?
Disallow: /memberlist.php
Disallow: /search.php?
Disallow: /index.php?

I am looking over my crawl errors in the google webmaster tools and here is my robots.txt file:

Code: Select all

User-agent: *
Disallow: /adm/
Disallow: /ads/
Disallow: /antispam/
Disallow: /blocks/
Disallow: /cache/
Disallow: /dl_mod/
Disallow: /docs/
Disallow: /download/
Disallow: /files/
Disallow: /includes/
Disallow: /language/
Disallow: /phpbb_seo/
Disallow: /store/
Disallow: /styles/
Disallow: /umif/
Disallow: /umil/
Disallow: /common.php
Disallow: /config.php
Disallow: /mcp.php
Disallow: /memberlist.php
Disallow: /posting.php
Disallow: /report.php
Disallow: /search.php
Disallow: /ucp.php


Why are they reporting that they are having crawling errors to memberlist, ucp, ads, search and download pages when they are supposed to be restricted from them in the first ...

If i block my /style/ directory from all the bots will that be wise like?
Disallow: /styles/

I recently installed phpbb3 (first time) and everything seems fine except that when making the site map, the same topic is crawled 5 times. After modifying the robots.txt using
User-agent: *


Disallow: /forum/phpBB3/viewtopic.php?p=
Disallow: /forum/phpBB3/viewtopic.php?=&p=
Disallow: /forum/phpBB3/viewtopic.php?t=
Disallow: /forum/phpBB3/viewtopic.php?start=
Disallow: /forum/phpBB3/*&view=previous
Disallow: /forum/phpBB3/*&view=next
Disallow: /forum/phpBB3/*&sid=
Disallow: /forum/phpBB3/*&p=
Disallow: /forum/phpBB3/*&sd=a

It has reduce to 4 times per topic. (only "viewtopic.php?p=" is disallowed) I notice that files having /*&sid=, /*&sd=a, and start=, are not blocked. Puzzling though, is that if I substitute the p for f as in "viewtopic.php?f", everything is disallowed. Can you help please?

Still somewhat new to working on the robots.txt file and sometimes I don't even think it works? I add profile.php to the list and then I observe a bot looking at the profiles :)

I know the system isn't perfect but is there a standard format for phpbb3.01 that provides the best security for users on the forums but also make the site indexable?

Can someone post an example of a robot file that maximizes user/site security but at the same time maximizes index ability of your forums?


Thanks

-D1-