Blocking all other bots except google,msn,yahoo

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Moderator: Moderators


Blocking all other bots except google,msn,yahoo

Postby kemraj » Mon Jun 15, 2009 11:44 pm

How to block all other bots except the major bots as said in the title while blocking them from the basic files
Disallow: /viewtopic.php?
Disallow: /viewforum.php?
Disallow: /groupcp.php
Disallow: /profile.php?
Disallow: /memberlist.php
Disallow: /search.php?
Disallow: /index.php?
kemraj
 
Posts: 32
Joined: Thu Mar 05, 2009 12:58 am

Advertisement

Re: Blocking all other bots except google,msn,yahoo

Postby dcz » Wed Jun 17, 2009 10:00 am

You can specify the list of bots that should not access your page in your robots.txt, but since only the good ones do more or less follow the robots.txt disallow, it's pretty much useless to disallow other like this.

Using an .htaccess based method is stronger, but still, if it relies on the UA, it's not absolute, since the UA is defined by the client.

So I would personally not bother too much about this.

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 19816
Joined: Fri Apr 28, 2006 9:03 pm

Re: Blocking all other bots except google,msn,yahoo

Postby HB » Wed Jun 17, 2009 2:51 pm

kemraj wrote:How to block all other bots except the major bots as said in the title while blocking them from the basic files

Suggestions from Google:

http://www.google.com/support/webmaster ... swer=40364

Code: Select all
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /


Just an example. It's easy enough to do with a combination of robots.txt, .htaccess, or the ACL robots page. This of course begs the question... why block?
Dan Kehn
HB
phpBB SEO Team
phpBB SEO Team
 
Posts: 1106
Joined: Mon Oct 16, 2006 2:25 am

Re: Blocking all other bots except google,msn,yahoo

Postby kemraj » Fri Jul 31, 2009 5:56 pm

I'm on a shared server so i don't what unnecessary server loads and bandwidth loss.
kemraj
 
Posts: 32
Joined: Thu Mar 05, 2009 12:58 am

Re: Blocking all other bots except google,msn,yahoo

Postby dcz » Mon Aug 03, 2009 12:32 pm

Problem is that only the "good" bots (eg the major search engine ones) will more or less follow the robots.txt directives, all the one you could want to filter here because they are not friendly or because the cost a lot of bandwidth will most likely do not care about the robots.txt.

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 19816
Joined: Fri Apr 28, 2006 9:03 pm


Return to roBots

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 1 guest