Googlebot

[legal]

And another question please.


How often will Googlebot typically crawl a site?

[dcz]

Well this depends on the number of pages and the PageRank of a web site.

And of course if your site is spider-able.

Then, the principle is the following, the more PageRank the more visits, and this is proportional to your number of pages but under certain circumstances.

For example, if you do not have cache-able duplicates (meaning the few ones are disallowed by the robots.txt) then this proportionality is only moderate by the PageRank Transmission, since pages inside your domain with good PageRank will of course be crawled more often than the other ones.

That's why new content listing is important as it will help bots to find new links even though they do not have PageRank. As well our best content should be the most linked.

Then, is you do have a lot of cache-able duplicates, then, the number of page will not increase bots visits as much, since PageRank will be diluted a lot and many cached page could end up in the Google omitted results.

As well, but this is linked to PageRank, the more links to your domain and content, the more visits.

So there is not really an average to wait for, it depends. What is for sure is an important step is reached when Google start loading more than one URL a day and better if they are distinct.

robostats is a good tool to track bot visits

++

[acapellas4u]

[dcz]

Well, taking an eye on bots visits is only useful to make sure your pages are crawled correctly, once you're sure about this, it's just not this important to track them anymore.

Robostat is using php/mysql and is thus less performant than Apache logs, but is a really handy tool when you start SEOing your web-site as it will tell you all the infos you need to know everything about bots visits at once.

I did not tested awstats, used Urchin and webalizer. Urchin does provide as well really accurate info about bots visit, but I find the robotstats output more user friendly in the end as it's a dedicated tool, performing real good synthesis.

But, on a big forum, and once you're sure to be correctly crawled, then uninstalling robotstat can be a good thing to lower server load, since anyway those infos are not that useful anymore (is my site crawl-able is the answer we want from such tools).

++