First rules for robots.txt SEO friendly?

Darth Pincho · PR1 Joined: 22 Sep 2006 Posts: 141

Hi i wan to know wich are the first things i have to edit in my robots.txt file in order to make my phpbb forum firendly?

I have no clue for that...

I will really appreciate some suggestions.

Thanks

dcz

The robots.txt content depends on you site and your URLs.

As a starting principle you should disallow any page being a duplicate of what you consider as the best URL to offer for each page, obviously if your script outputs more than one kind.

As well, files only meant for example to post messages should be disallowed in order to help bots focusing on your real content, the result of those posts if we follow this example.

++

Darth Pincho · PR1 Joined: 22 Sep 2006 Posts: 141

Here you have the complete robots.txt file at my site

dcz

If you had this robots.txt before you found out about this site, then it's very normal you where not spidered, you where disallowing all topics and posts Wink

So keep this robots.txt and add the line suggested in the release topic of the mod rewrite you are using Wink

++

Darth Pincho · PR1 Joined: 22 Sep 2006 Posts: 141

No friend. This is my new robots file after i install mod rewrite mixed.

Because of that I disallow all the php extensions in order to not have duplicates.

In this case is ok? or i have to change something?

Peter77

Your robots txt is fine. also you may add any other files you do not want search engines to crawl - like any MODs you might have. shop.php, album.php, rss.php ect. Cool

dcz

I already told you to get rid of this part :