How to block all other bots except the major bots as said in the title while blocking them from the basic files
Disallow: /viewtopic.php?
Disallow: /viewforum.php?
Disallow: /groupcp.php
Disallow: /profile.php?
Disallow: /memberlist.php
Disallow: /search.php?
Disallow: /index.php?

You can specify the list of bots that should not access your page in your robots.txt, but since only the good ones do more or less follow the robots.txt disallow, it's pretty much useless to disallow other like this.

Using an .htaccess based method is stronger, but still, if it relies on the UA, it's not absolute, since the UA is defined by the client.

So I would personally not bother too much about this.

++

kemraj wrote:How to block all other bots except the major bots as said in the title while blocking them from the basic files

Suggestions from Google:

http://www.google.com/support/webmaster ... swer=40364

Code: Select all

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /

Just an example. It's easy enough to do with a combination of robots.txt, .htaccess, or the ACL robots page. This of course begs the question... why block?

I'm on a shared server so i don't what unnecessary server loads and bandwidth loss.

Problem is that only the "good" bots (eg the major search engine ones) will more or less follow the robots.txt directives, all the one you could want to filter here because they are not friendly or because the cost a lot of bandwidth will most likely do not care about the robots.txt.

++