My installed mods include:
- advanced mod rewrite
- dynamic meta tags
- advanced zero duplicates

What is a recommended robots.txt file? Until now, I am using this:

Code: Select all

User-agent: *
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /privmsg.php
Disallow: /post
Disallow: /member
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php

Is this good? Should I change it? I want maximum SEO

i think your robots.txt file are good. just one little thing
you have installed zero dupe. you can remove

Code: Select all

Disallow: /post

from your robots.txt file because zero dupe will redirect all post URL to proper topic URL

Thank you for your reply.

Shouldn't a robots.txt file block this sort of thing?

http : // www . hvacrunivers . com / probe . php ? extra=872a3e51e4ce4ac57bcc72afd2f4eb92,4b3e6ab3c7

+ADw-body+AD4 +ADw-iframe src+AD0AIg-http : // www . hvacrunivers . com ...

What do you mean ?

The robots.txt does not actually block anything, it's just a set of directive we ask bots to follow, in the end it's up to them to be nice or not.

++

dcz wrote:What do you mean ?

The robots.txt does not actually block anything, it's just a set of directive we ask bots to follow, in the end it's up to them to be nice or not.

++

OK, I'll elaborate. What I mean is ... Is there any reason why I would want google to go through files such as probe.php? Should I be adding more files to my robots.txt file to prevent indexing of such files as this?

I don't recognize probe.php, but as a general statement, robots.txt lists files / directories that you don't want indexed, e.g.:

User-agent: *
Disallow: /forums/admin/
Disallow: /forums/images/
Disallow: /forums/cache/
Disallow: /forums/db/
Disallow: /forums/includes/
Disallow: /forums/language/
Disallow: /forums/templates/
Disallow: /forums/groupcp.php
Disallow: /forums/memberlist.php
Disallow: /forums/faq.php
Disallow: /forums/login.php
Disallow: /forums/modcp.php
Disallow: /forums/posting.php
Disallow: /forums/search.php
Disallow: /forums/profile.php
Disallow: /forums/privmsg.php
Disallow: /forums/viewonline.php

The robots.txt entries are "hints" that save the bot time and your server resources. It's not an access list; for that, you would use .htaccess / .htpasswd

HB wrote:The robots.txt entries are "hints" that save the bot time and your server resources. It's not an access list; for that, you would use .htaccess / .htpasswd

Thank you for your reply. The thing is I don't know what is best to allow/disallow via a robots.txt file. I used a recommended one off this site and I assumed it would "direct" the robots into the proper places. So, I guess I am simply wondering if there are more files/directories that I should include in my robots.txt file. Perhaps, the better question is "what files or directories should I be sure to include so that the search engines will go to relevant areas for indexing?" I guess I am looking for a comprehensive answer on a well-formed robots.txt file that "asks" the bots not to go to unnecessary places. Thank you.

Assuming you're using GYM sitemap, you've got all the entry points covered for your forum. From the robots.txt viewpoint, the only question is what you DON'T want indexed. To the best of my knowledge, the settings above are exhaustive for the standard phpBB install. You may wish to check google's index to confirm extraneous pages aren't included.

FYI, google's webmaster tools now includes removal requests and it accepts wildcards if you wish to tidy up past mistakes. Whether it matters SEO-wise, I don't know, but it helps googlebot's efficiency.