| :: |
| Author |
Message |
Orbits
Joined: 16 Jun 2007
Posts: 35
|
 Posted: Wed Jul 25, 2007 7:31 pm Post subject: googlebot crawl rate slowed day after GYM sitemaps installed |
|
|
I have several phpBB Forum's which I manage. Some are really big and some are really small.
I track all traffic to the site using the Google Analytics as well as my own log scrapping app. awstats.
BEFORE
Before I notified GOOGLE about my sitemap I usually had around 200-500 pages indexed per day from the googlebot crawler (that's my approx. new post count on one of my forums). Yahoo/Slurp did about 50 pages per day (or so).
AFTER
After I notified google about my sitemaps, i'm only getting about 5-10 pages crawled by the googlebot per day now. Yahoo/SLURP just loves the sitemap however and they are now busy crawling my entire site daily pull hundreds and sometimes thousands of pages per day.
Thoughts?
SteveFree |
Last edited by Orbits on Wed Oct 15, 2008 7:07 am; edited 1 time in total |
|
| Back to top |
|
 |
|
|
HB
phpBB SEO Team
Joined: 16 Oct 2006
Posts: 831
|
| Posted: Fri Jul 27, 2007 4:34 pm Post subject: Re: googlebot crawl rate slowed day after GYM sitemaps installed |
|
|
What is your sitemap URL? FYI, the new sitemap protocol defines entries for robots.txt:
| Quote: | | Sitemap:http://www.yourdomain.com/forums/sitemaps.xml |
|
_________________
Dan Kehn |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 15242
|
| Posted: Fri Jul 27, 2007 6:27 pm Post subject: Re: googlebot crawl rate slowed day after GYM sitemaps installed |
|
|
Well 4 indexed pages is not really compatible with 500 crawled pages a day from Google.
Same for the main domain : http://www.google.com/search?q=site%3Awww.groovypost.com
Beside, I could not find any sitemap.php there : -http://forum.groovypost.com/sitemap.php
And, bot visit is not exactly the same thing as cached page. With the whole collection of phpBB duplicate, bots are losing a lot of time to find new content, and visit many time the same page for nothing. GYM sitemaps helps them to directly target the good ones.
Playing with settings should allow you to list all the threads of your biggest forum, so far, we tested with up to 17 000 urls in a single sitemaps without difficulties.
What could be more efficient to provide bots with your best urls ?
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
Orbits
Joined: 16 Jun 2007
Posts: 35
|
| Posted: Sun Sep 09, 2007 5:27 am Post subject: Re: googlebot crawl rate slowed day after GYM sitemaps installed |
|
|
| Sorry, the site I applied the MIXED SEO to is http://www.dogs4sale.net |
Last edited by Orbits on Wed Oct 15, 2008 7:09 am; edited 1 time in total |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 15242
|
| Posted: Mon Sep 10, 2007 8:42 am Post subject: Re: googlebot crawl rate slowed day after GYM sitemaps installed |
|
|
1410 cached page : http://www.google.com/search?q=site%3Awww.dogs4sale.net
So, when migrating / installing the mod, it's normal to see fewer pages listed, since we get rid of all duplicates and they can be many up to over 20 per page.
So this artificially increase the crawling but at the same time, lowers the interest of the site, since many visits are useless (same exact content).
The Google sitemap system as well can lower a bit the crawling, since it allow bots to do a lot more with fewer visits (there is no duplicate listed there, a single visit is required to cache the listed pages).
But this implies that your sitemap is working, and it does not seems to be the case : -http://www.dogs4sale.net/sitemaps.xml is currently unreachable.
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
Orbits
Joined: 16 Jun 2007
Posts: 35
|
| Posted: Tue Sep 11, 2007 3:57 am Post subject: Re: googlebot crawl rate slowed day after GYM sitemaps installed |
|
|
| dcz wrote: |
The Google sitemap system as well can lower a bit the crawling, since it allow bots to do a lot more with fewer visits (there is no duplicate listed there, a single visit is required to cache the listed pages).
But this implies that your sitemap is working, and it does not seems to be the case : -http://www.dogs4sale.net/sitemaps.xml is currently unreachable.
++ |
YIKES,
typo in one of my redirects I made yesterday! Thanks for the FYI, I wouldn't have noticed it!
Thanks
-Steve |
|
|
| Back to top |
|
|
HB
phpBB SEO Team
Joined: 16 Oct 2006
Posts: 831
|
| Posted: Tue Sep 11, 2007 7:56 pm Post subject: Re: googlebot crawl rate slowed day after GYM sitemaps insta |
|
|
On a related note...
| stevefree wrote: | | Yahoo/SLURP just loves the sitemap however and they are now busy crawling my entire site daily pull hundreds and sometimes thousands of pages per day. |
I don't know what Yahoo/SLURP is smoking these days (re-reading the same content THOUSANDS of times), but it finally wore out my patience and I added this to robots.txt:
| Code: | User-agent: Slurp
Crawl-delay: 30 |
This instructs Slurp to "chill out" for 30 seconds between queries. I noticed an improvement in the site's performance, and no change in the site's laughably small traffic from Yahoo. Another to consider is this:
| Code: | User-agent: Fasterfox
Disallow: / |
It disables the Firefox plug-in Fasterfox from pre-fetching your site's pages. |
_________________
Dan Kehn |
|
| Back to top |
|
|
|
|
