Robots/Spiders visitors

londonita · Joined: 20 Dec 2006 Posts: 1 Location: London

Hi guys
my site get visited by 13 robots
Inktomi Slurp,Googlebot,MSNBot,Alexa,EchO!,AskJeeves,LinkWalker,Voyager,WISENutbot plus other 4 Unknown robots. Around 9 of them make a daily visit.
My question is : for doing very well, how many robots should visit your site?
Do anyone know when there is the next GOogle Dance?
Any feed back is much appreciated
Thank you

dcz · Posted: Wed Dec 20, 2006 7:50 pm Post subject: Re: Robots/Spiders visitors

And welcome Very Happy

Well, there is not really a specified number of bots visits to reach, it's relative.

It depends on your web-site's number of spiderable pages and the way new pages are added or content updated.
It depends as well on how well your web-site is ranked and on the number of backlinks.

The more spiderable pages, the more content added and updated on a regular basis, the more backlinks, and more bots visits and better racking.

And "spiderable page" can mean a bit more than just public page, you'll get very different results with clean mod rewritten URLs and no dupes than with just regular dynamic URLs with a bunch of vars.

In the end, time is as well something that matter, with time a web-site will be visited by more and more bots, there are plenty of many different kinds, friendly or not.

So if we try to set up some kind of a criteria, I think a good thing to track is the time needed for a full cache update. Basically, if you track bots visits and keep track of visited URLs, you should be able to tell if all your pages are crawled and how long does it takes more or less to see as many URLs spidered as your site's have for each bots.

For example so far, this web site get fully crawled every week or so by Google, which seems like a real good rate.

Crawltrack is doing some useful stats about these visits.

++