I found some good php based software for indexing bots that come on your site:
www.crawltrack.info
Its shows Google, Yahoo, MSN and some other bots.

Implemenation into phpBB:
1. Click on settings
2. Go to "Display tags to insert"
3. Choose the one with no logo- it could cause some problems.
4. Youll get a piece of code that you need to add in an php file.
5. Open includes/page_header.php
6. Find:

Code: Select all

?>

Add your code before.

List of some bots supported by CrawlTrack:

AbachoBot, ABCdatos, Acoon Robot, Aesop, Aibot, Alexa, Altavista, Amfibibot, Amidalla, Antibot, ArchitextSpider, Ask Jeeves/Teoma, Baiduspider, Blitzsuche, Blogbot, Boitho, Bruinbot, CipinetBot, Clushbot, Cobion, Cortina, DataFountains, Drecombot,Earthcom, Elsop, EuripBot, Exabot, FAST-WebCrawler, GenieKnows, GeonaBot, Goblin, Google-Adsense, Googlebot, Google-Image, Google-WAP, Heritrix, InelaBot, Jayde Crawler, LinkWalker, Lockstep Spider, Lycos_Spider, Mariner, Mercator, MSN Bot, Najdi, NaverBot, NokodoBot, OpidooBot, OpenWebSpider, Polybot, Pompos, Psbot, QuepasaCreep, Scrubby, Seekbot, Slurp Inktomi (Yahoo), Teoma, Toutatis, TygoBot, VoilaBot, WiseGuys, Zao Crawler, ZyBorg (LookSmart)

Interesting, thanks for sharing

Seems even better than robotstats.

Good thing some one decided to dev and maintain such software

++

Another simple way to track googlebot is to ad the following code to the bottom of common.php

BEFORE

Code: Select all

?>

ADD

Code: Select all


if ($HTTP_SERVER_VARS["HTTP_X_FORWARDED_FOR"]!= ""){
$host = @gethostbyaddr($HTTP_SERVER_VARS["HTTP_X_FORWARDED_FOR"]);
}else{
$IP = $HTTP_SERVER_VARS["REMOTE_ADDR"];
$host = @gethostbyaddr($HTTP_SERVER_VARS["REMOTE_ADDR"]);
}
if(eregi("googlebot",$host))
{
$uri = "http://www.YOURURL.com" . $_SERVER['REQUEST_URI'];
$ip = $_SERVER['REMOTE_ADDR'];
$ref = $_SERVER['HTTP_REFERER'];
$dtime = date('r');
$unixtime=time();
$agent1 = $_SERVER["HTTP_USER_AGENT"];


$line = "$unixtime;$dtime;$ip;$uri;$ref;$host;$agent1";

$f = fopen("bots.txt", "a");

@fwrite($f, $line . "\n");
@fclose($f);
}


Simple indeed.

The other two will perform some interesting stats without being this heavier.

And they'll track more bots and are less likely to be tricked by user faking User Agent, which I admit is not the often, as they as well track IPs.

But sometime, simplicity is good. It's just you probably will have some performances issues after you reach too many visitors and get crawled too much.
What if 50 Googlebots visit your site at the same time ? Mysql can handle zillion updates at a time where direct php file access will only allow for one at a time.

++

Seems interesting but can someone elaborate on what this does exactly and how can this be useful for people like me.

It's useful to track how bots are visiting your web-site : number of page loaded and which ones with detailed statistics.

It's not like a must once you know you're crawled well, and even a bit less useful for Google since the Google sitemap User Panel is now providing some statistics about Google-Bot spidering.
But, it's still interesting, and even more when you're not already crawled to death, to have these datas.

It's a good way to find out robots.tx mistakes, as you can list the URL crawled, and to get and idea about your web-site's relationship with bots.

Obviously, this does not apply to euroman's script, which is far simpler than CrawlTrack.

++

Seems Cool !!!

I more interested to know about Yahoo bot visiting ( or not visiting ) my site bcos it has been dropped from number 1 position to nowhere on the first page now.
I hope this thing can help me in knowing which pages yahoo bot is spidering and at what frequency. It wud be great if there is some software that can evaluate the reason of a website being at number 1 position in google and nowhere near the site in yahoo

Yahoo! just need more time than Google to have a web site well ranked, especially if you don't subscribe to their directory.

But with time, if it's properly showing up in Google, it will in yahoo! as well.

++

My website was ranked at number 1 position for several keywords on Yahoo for almost 5 years but since last one year it has gone way down. My website is listed in yahoo diretory Yahoo Directory

It was a good idea then to start a forum and to Search Engine Optimize it, because it will make your site more dynamic, with new content added more often.

We are going off topic, but I would take this as an occasion to start spending more time on my website SEO and development.
Think about how you could continue to keep it growing with good content and to work on obtaining more Backlinks in related website or directories.
As well, you need to study your challengers a bit, not to copy (I really think each website should be the more specific and unique as possible in content and design), but to know who is now showing up first on your interesting search queries and set up an appropriate strategy.
The idea is to work on what is already working the best with your project while still spending time wondering what could be added to make it even better. Industry and research

++

No I think we are on topic only, bcos its not about my website but about studying the way different robots index same website.
Googlebot ranks me at number 1 position for around 10 keywords but yahoo and MSN doesnt seem to agree with google and rank my website at much lower position. As far as I am concerned it doesnt matter much to me till google ranks me at number 1 position.

But this behaviour of robots is worth studying.

I feel it would change in times to come as Yahoo and MSN have agreed to use same sitemap protocol as Google does

Oh, ok, I though you where wondering about the possible causes for this drop.

And yes, the Sitemap United Protocol is a really good thing to help out indexing and page caching. It should do as it did with Google, less visits needed to spider a web-site, thus more cached pages.

++

Yeah. I hope we can expect some major changes in the way pages are indexed by other search engines now. Now there will be unanimity on large scale on the way SEs index our pages.

Not this sure about it. I mean sitemap is a great tool to help out bots spidering webs sites, but they still need to do it. If we look at how much time is required before Yahoo! start indexing more than the home page of a new web-site, I am not sure it will drastically change, unless maybe you pay the directory submission ...
what is for sure is the job will be easier for them with sitemaps, let's hope easier mean better here.

++

euroman wrote:Another simple way to track googlebot is to ad the following code to the bottom of common.php

BEFORE

Code: Select all

?>

ADD

Code: Select all


if ($HTTP_SERVER_VARS["HTTP_X_FORWARDED_FOR"]!= ""){
$host = @gethostbyaddr($HTTP_SERVER_VARS["HTTP_X_FORWARDED_FOR"]);
}else{
$IP = $HTTP_SERVER_VARS["REMOTE_ADDR"];
$host = @gethostbyaddr($HTTP_SERVER_VARS["REMOTE_ADDR"]);
}
if(eregi("googlebot",$host))
{
$uri = "http://www.YOURURL.com" . $_SERVER['REQUEST_URI'];
$ip = $_SERVER['REMOTE_ADDR'];
$ref = $_SERVER['HTTP_REFERER'];
$dtime = date('r');
$unixtime=time();
$agent1 = $_SERVER["HTTP_USER_AGENT"];


$line = "$unixtime;$dtime;$ip;$uri;$ref;$host;$agent1";

$f = fopen("bots.txt", "a");

@fwrite($f, $line . "\n");
@fclose($f);
}


who use this and does it useful