Possible for search engine bot to crawl more than 24 hours?

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Moderator: Moderators

Possible for search engine bot to crawl more than 24 hours?

Postby Peter77 » Sun Jun 18, 2006 8:17 pm

Anyone know if it's normal to have a search engine bot stay on your site for many hours? I have a Ask Jeeves bot that has been on my site for almost 24 hours now. is it stuck, or can this be normal for a search engine bot to stay so long? RobotStats does not even document it even though the IP's match. at first I thought it just kept visiting... but now I think it's just "parked" on my site.

any ideas?

Thanks

::edit::
Wow, okay it seems as though it finally left my site. but my question still remains.. is this normal for search engine bots to stay so long? and what are they actually doing that they stick around so long... do some search egine bots spider a lot 'slower'?
Peter77
phpBB SEO Team
phpBB SEO Team
 
Posts: 524
Joined: Wed May 10, 2006 9:46 am
Location: Michigan

Advertisement

Postby dcz » Sun Jun 18, 2006 11:12 pm

Well, many things can cause bots to stay a long time on your web-site.
It's most likely that Ask Jeeves either spidered all of your site for the first time or just updated his cache. Many pages take some time to spider so, 24 hours is not that much to me.

Do you know if you where indexed in ask Jeeves before this ?

Anyway with the recent changes you implemented, it's normal that most bots will like your site better and go through a total re-index, new URLs, new meta tags, new titles, new sitemaps, yes it makes some difference :D

The important thing here is to watch for bot either not following the robots.txt or spidering way to fast, but Ask Jeeves is a good one, so.

This just mean your site is really online, and bot friendly, so they show up ;)
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21238
Joined: Fri Apr 28, 2006 9:03 pm

Postby Peter77 » Wed Jun 21, 2006 4:14 am

Oh yes, I've been indexed by Ask Jeeves a while back. i've seen it visit a few times, but never actually stay so long and make so many visits like it has been. :D thanks for the info.
Peter77
phpBB SEO Team
phpBB SEO Team
 
Posts: 524
Joined: Wed May 10, 2006 9:46 am
Location: Michigan

Postby dcz » Wed Jun 21, 2006 8:40 am

So that's probably how much more Rewritten links are loved by bots ;)

And that's for sure linked to the fact PageRank is better transmitted when using mod Rewrite, better links, better crawls ....

To stay this long, Ask Jeeves must have gone through some updating in his cache DB, after you have changed so much things, and actually this is a good sign :D

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21238
Joined: Fri Apr 28, 2006 9:03 pm


Return to roBots

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 1 guest