Possible for search engine bot to crawl more than 24 hours?

Peter77

Anyone know if it's normal to have a search engine bot stay on your site for many hours? I have a Ask Jeeves bot that has been on my site for almost 24 hours now. is it stuck, or can this be normal for a search engine bot to stay so long? RobotStats does not even document it even though the IP's match. at first I thought it just kept visiting... but now I think it's just "parked" on my site.

any ideas?

Thanks

::edit::
Wow, okay it seems as though it finally left my site. but my question still remains.. is this normal for search engine bots to stay so long? and what are they actually doing that they stick around so long... do some search egine bots spider a lot 'slower'?

dcz

Well, many things can cause bots to stay a long time on your web-site.
It's most likely that Ask Jeeves either spidered all of your site for the first time or just updated his cache. Many pages take some time to spider so, 24 hours is not that much to me.

Do you know if you where indexed in ask Jeeves before this ?

Anyway with the recent changes you implemented, it's normal that most bots will like your site better and go through a total re-index, new URLs, new meta tags, new titles, new sitemaps, yes it makes some difference Very Happy

The important thing here is to watch for bot either not following the robots.txt or spidering way to fast, but Ask Jeeves is a good one, so.

This just mean your site is really online, and bot friendly, so they show up Wink

Peter77

Oh yes, I've been indexed by Ask Jeeves a while back. i've seen it visit a few times, but never actually stay so long and make so many visits like it has been. Very Happy

thanks for the info.

dcz

So that's probably how much more Rewritten links are loved by bots Wink

And that's for sure linked to the fact PageRank is better transmitted when using mod Rewrite, better links, better crawls ....

To stay this long, Ask Jeeves must have gone through some updating in his cache DB, after you have changed so much things, and actually this is a good sign Very Happy

++