| |
| |
|
|
|
|
| |
|
| |
|
| :: |
| Author |
Message |
zinnerz
PR0
Joined: 08 Jun 2006
Posts: 75
|
 Posted: Sat Feb 10, 2007 11:27 pm Post subject: Review my robots.txt |
|
|
Dear all,
Please review my robots.txt, is this ok?
| Code: | User-agent: googlebot
Disallow:
Disallow: *.php$
Disallow: post
Disallow: /*?
Disallow: /*?*
User-agent: slurp
Crawl-delay: 15
Disallow:
Disallow: /*.php$
Disallow: /post
Disallow: /*?
Disallow: /*?*
User-agent: msnbot
Crawl-delay: 15
Disallow:
Disallow: /*.php$
Disallow: /post
Disallow: /*?
Disallow: /*?*
User-agent: Googlebot-Image
Disallow:
User-Agent: MediaPartners-Google
Disallow:
User-agent: *
Disallow: / |
My forum is in the root  |
|
|
| Back to top |
|
 |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 13354
|
| Posted: Sun Feb 11, 2007 12:20 pm Post subject: Re: Review my robots.txt |
|
|
| Code: | User-agent: *
Disallow: / |
Is basically disallowing everything for all bots.
| Code: |
User-Agent: MediaPartners-Google
Disallow: |
Is a mistake is you are using AdSense, and is useless if not (the media partner bots does not really crawl unless you run adds).
Disallows with wild-cards are not really reliable.
According to what I see you tried to do, I'd only keep :
| Code: | User-agent: *
Crawl-delay: 15
|
and then, as you're talking about a forum, it depends on you urls standard, if it's phpBB and running a phpBB SEO mod rewrite, then, you should just add the disallow advised in the release thread, if not, you'd still need to add some disallow to prevent duplicates.
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
zinnerz
PR0
Joined: 08 Jun 2006
Posts: 75
|
| Posted: Sun Feb 11, 2007 12:43 pm Post subject: Re: Review my robots.txt |
|
|
| dcz wrote: | | Code: | User-agent: *
Disallow: / |
Is basically disallowing everything for all bots. |
I dont know about this as 3 bots already allowed above it, so far I've seen in Show Bots mod and View Online IP Address, the 3 bots is still crawling and everyday the numbers of list is increasing.
| Quote: | | Code: |
User-Agent: MediaPartners-Google
Disallow: |
Is a mistake is you are using AdSense, and is useless if not (the media partner bots does not really crawl unless you run adds). |
I will remove it
| Quote: | | Disallows with wild-cards are not really reliable. |
I want to test it and see what happens, hopefully they are working.
| Quote: | and then, as you're talking about a forum, it depends on you urls standard, if it's phpBB and running a phpBB SEO mod rewrite, then, you should just add the disallow advised in the release thread, if not, you'd still need to add some disallow to prevent duplicates.
++ |
I will update here again after 1 week, see are they crawling or not .php and post.html
Thanks Dcz for your review
SS:
 |
Last edited by zinnerz on Sun Feb 11, 2007 1:10 pm; edited 1 time in total |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 13354
|
| Posted: Sun Feb 11, 2007 1:07 pm Post subject: Re: Review my robots.txt |
|
|
Well, it's a bit different to see bots on your forum, than having them actually cache pages.
As it's phpbb, my advice is in your case :
| Code: | User-agent: *
Crawl-delay: 15
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /privmsg.php
Disallow: /post
Disallow: /member
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php |
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
Ľubor
PR0
Joined: 13 Feb 2007
Posts: 52
Location: Slovakia,Bratislava
|
| Posted: Mon Mar 12, 2007 5:23 pm Post subject: Re: Review my robots.txt |
|
|
| Code: | User-agent: *
Disallow: viewtopic.php
Disallow: viewforum.php
Disallow: index.php?
Disallow: posting.php
Disallow: groupcp.php
Disallow: search.php
Disallow: login.php
Disallow: privmsg.php
Disallow: profile.php
Disallow: post
Disallow: member |
is good?
http://pc-war.sk/robots.txt |
Last edited by Ľubor on Mon Mar 12, 2007 7:52 pm; edited 1 time in total |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 13354
|
|
| Back to top |
|
|
|
|
| Navigation |
Similar Topics |
|
|
|
|
|
|
|
| |
|
|
|
|
| |
|
|
|
|
| |
