Dear all,

Please review my robots.txt, is this ok?

Code: Select all

User-agent: googlebot
Disallow:
Disallow: *.php$
Disallow: post
Disallow: /*?
Disallow: /*?*

User-agent: slurp
Crawl-delay: 15
Disallow:
Disallow: /*.php$
Disallow: /post
Disallow: /*?
Disallow: /*?*

User-agent: msnbot
Crawl-delay: 15
Disallow:
Disallow: /*.php$
Disallow: /post
Disallow: /*?
Disallow: /*?*

User-agent: Googlebot-Image
Disallow:

User-Agent: MediaPartners-Google
Disallow:

User-agent: *
Disallow: /

My forum is in the root

Code: Select all

User-agent: *
Disallow: /

Is basically disallowing everything for all bots.

Code: Select all

User-Agent: MediaPartners-Google
Disallow:

Is a mistake is you are using AdSense, and is useless if not (the media partner bots does not really crawl unless you run adds).

Disallows with wild-cards are not really reliable.

According to what I see you tried to do, I'd only keep :

Code: Select all

User-agent: *
Crawl-delay: 15


and then, as you're talking about a forum, it depends on you urls standard, if it's phpBB and running a phpBB SEO mod rewrite, then, you should just add the disallow advised in the release thread, if not, you'd still need to add some disallow to prevent duplicates.

++

dcz wrote:

Code: Select all

User-agent: *
Disallow: /

Is basically disallowing everything for all bots.

I dont know about this as 3 bots already allowed above it, so far I've seen in Show Bots mod and View Online IP Address, the 3 bots is still crawling and everyday the numbers of list is increasing.

Code: Select all

User-Agent: MediaPartners-Google
Disallow:

Is a mistake is you are using AdSense, and is useless if not (the media partner bots does not really crawl unless you run adds).

I will remove it

Disallows with wild-cards are not really reliable.

I want to test it and see what happens, hopefully they are working.

and then, as you're talking about a forum, it depends on you urls standard, if it's phpBB and running a phpBB SEO mod rewrite, then, you should just add the disallow advised in the release thread, if not, you'd still need to add some disallow to prevent duplicates.

++

I will update here again after 1 week, see are they crawling or not .php and post.html

Thanks Dcz for your review

SS:

Well, it's a bit different to see bots on your forum, than having them actually cache pages.

As it's phpbb, my advice is in your case :

Code: Select all

User-agent: *
Crawl-delay: 15

Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /privmsg.php
Disallow: /post
Disallow: /member
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php

++

Code: Select all

User-agent: *

Disallow: viewtopic.php
Disallow: viewforum.php
Disallow: index.php?
Disallow: posting.php
Disallow: groupcp.php
Disallow: search.php
Disallow: login.php
Disallow: privmsg.php
Disallow: profile.php
Disallow: post
Disallow: member

is good?
http://pc-war.sk/robots.txt

Yes the robots.txt is correct, but you should install the zero duplicate, would get rid of all SID duplicate in SE's listing

++