I installed the simple showbots a few days ago and it seems to work well

Each time I check my forum there is always a huge amount of Yahoo bots online, there is 28 Yahoo bots online right now

Is this normal and does it mean anything, apart from the fact that Yahoo have lots of bots?

Thanks in advance

chez

Yes, yahoo is really the most aggressive bot compared to Google's and MSN's.

It's often crawling using many different ips and thus creating the same amount of sessions. Google crawls a comparable amount of pages but mostly uses an ip at a time.
I don't really know why yahoo uses so many ips at a time, and what could be their interest in doing so, but that's how yahoo crawls, many ip, many pages loads at a time.

The funny thing on phpBB SEO is Yahoo bots comes more often than it brings visitors


++

dcz wrote:
The funny thing on phpBB SEO is Yahoo bots comes more often than it brings visitors


++

in my site the same too

I get tons of people on my site

Mainly bloomin spammers that post links to porn/illegal sites

It keeps me busy banning the little swines

I thus suggest this combination to fight against spammers :

sessions hash : http://www.phpbb.com/phpBB/viewtopic.php?t=430710
and Better captcha : http://www.phpbb.com/phpBB/viewtopic.php?t=473222

++

dcz wrote:The funny thing on phpBB SEO is Yahoo bots comes more often than it brings visitors

Same here. Google reads the sitemap without prompting, covers half the site in less than a day and is on its way. Yahoo continuously chews on pages, ignores the sitemap unless explicitly requested, and brings a fraction of the traffic.

I've considered banning their bot on principle alone.

Out of curiosity, I installed Simple Show Bots this morning. Yahoo's legion of bots are now revealed. Below are a couple nits:

Code: Select all

$lang['Bot_online'] = 'Bots: ';

Changed from "Online Bots :" because I've never heard of an offline bot. Minor correction for the extra space after the comma:

Code: Select all

   // www.phpBB-SEO.com SEO TOOLKIT BEGIN
   if ( !empty($bots_online) ) {
      foreach ( $bots_online as $bot => $bot_num) {
         $bot_cnt = ( $bot_num > 1) ? " ($bot_num)" : '';
         $online_botlist .= (($online_botlist!='') ? ', ' : '') . "<span ".$bot_style[$bot].">".$bot.$bot_cnt."</span>";
      }
   }
   // www.phpBB-SEO.com SEO TOOLKIT END

dcz wrote:I thus suggest this combination to fight against spammers :

sessions hash : http://www.phpbb.com/phpBB/viewtopic.php?t=430710
and Better captcha : http://www.phpbb.com/phpBB/viewtopic.php?t=473222

++

Thanks for the recommendations

HB wrote:

dcz wrote:The funny thing on phpBB SEO is Yahoo bots comes more often than it brings visitors

Same here. Google reads the sitemap without prompting, covers half the site in less than a day and is on its way. Yahoo continuously chews on pages, ignores the sitemap unless explicitly requested, and brings a fraction of the traffic.

I've considered banning their bot on principle alone.

I'm glad that you told me that!

I submit the sitemap to Google regularly, but I only submitted to Yahoo once, thinking that they would check the urllist like Google checks the sitemap. I have just logged into Yahoo and it was using the urllist from August 2006 At least they will have the up to date version now

Yahoo ....


Anyway, about Google, you don't need to resubmit the sitemap, Google will follow the updates really well by itself.

++

I have just had 169 Yahoo bots on my forum at the same time

I'm thinking of removing the simple show bots mod as it looks like more guests are browsing then, rather than just plain old bots!

Yes it really looks like Yahoo is going crazy since yesterday. I have seen it using 43 sessions here, and it must have gone further since we broke our online record :

Code: Select all

Most users ever online was 109 on Wed Jun 06, 2007 8:55 am

But it's actually :

Code: Select all

Most users ever online was 122 on Wed Jun 06, 2007 8:55 am

On the portal, the difference only come from the fact that the portal is installed on the other forum, and thus will update the record checking in all the french forum + portal and directories pages, where the on from the English board, the record will only be updated when someone actually load a page in the forum, and it's of course less pages than forum + portal + directories, and thus fewer probability to actually update the record.

Since many site are currently experiencing kind of a heavy crawl from yahoo, maybe we can start talking about a yahoo dance.
Because, it crawled all the page from phpBB SEO like three times in a row these past 48 hours.
And I'm pretty sure it will go back to a more reasonable rate soon.

++

I was wondering if it would be possible to make the online bots only visible to the admin and not the general forum?

This would make the forum look like it had lots of visitors, rather then bots?

Well, there would not be much sens for a cosmetic feature to be available for the admin only, beside filtering the output would mean an extra "if" for all sessions from all bots.

What you can do if you like is to only count one single time each bots.

In page_header.php, find :

Code: Select all

         $bot_cnt = ( $bot_num > 1) ? "($bot_num)" : '';
         $online_botlist .= (($online_botlist!='') ? ', ' : '') . "<span ".$bot_style[$bot].">$bot $bot_cnt</span>";

Replace with :

Code: Select all

         if  ( $bot_num > 1) {
            $guests_online = $guests_online + $bot_num - 1;
         }
         $online_botlist .= (($online_botlist!='') ? ', ' : '') . "<span ".$bot_style[$bot].">$bot</span>";

this way, if yahoo crawls with 100 different ips, it will only be listed once, like if there was 99 guest and one bot.

++

Thanks dcz,

I just changed that code and it works much better now.

Thanks very much