Do i still need to make an robots.txt

Discussions about the phpBB2 Forum. How to get the best from this powerful script.

Moderator: Moderators

Do i still need to make an robots.txt

Postby emielvenlo » Mon Feb 26, 2007 7:33 pm

On my forum: [MOD]link removed[/MOD] I have made the following mods:
Advanced mod_rewrite
zero dupe
optimal titles

Do i still need to make an robots.txt for instance to not let google index posts(dupe's from topic) and members??
User avatar
emielvenlo
PR0
PR0
 
Posts: 82
Joined: Sun Jan 21, 2007 10:01 am

Advertisement

Postby HB » Tue Feb 27, 2007 2:13 am

The zero dupe will redirect viewtopic.php / viewforum.php to the new URLs; you don't want to block search engine access. Below are robots.txt rules I recommend:

Code: Select all
User-agent: *
Disallow: /forums/groupcp.php
Disallow: /forums/memberlist.php
Disallow: /forums/member
Disallow: /forums/faq.php
Disallow: /forums/login.php
Disallow: /forums/modcp.php
Disallow: /forums/posting.php
Disallow: /forums/search.php
Disallow: /forums/profile.php
Disallow: /forums/privmsg.php
Disallow: /forums/viewonline.php


dcz's list is slightly different:

http://www.phpbb-seo.com/robots.txt

They're essentially the same except for index.php, viewtopic.php, and viewforum.php. I omitted them to allow the search engines a chance to be redirected.
Dan Kehn
HB
phpBB SEO Team
phpBB SEO Team
 
Posts: 1509
Joined: Mon Oct 16, 2006 2:25 am

Postby dcz » Tue Feb 27, 2007 1:45 pm

Yes, you should not disallow :

Code: Select all
Disallow: /boards/viewtopic.php
Disallow: /boards/viewforum.php
Disallow: /boards/index.php


when migrating urls, but this can help out once it's done. As these will prevent all the advanced pagination sorting options to be cached as well.

So while migrating urls, I'd say, get rid of these for couple month, and put them back after, when bots already stopped using them.

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21376
Joined: Fri Apr 28, 2006 9:03 pm

Postby emielvenlo » Tue Feb 27, 2007 2:13 pm

dcz wrote:Yes, you should not disallow :

Code: Select all
Disallow: /boards/viewtopic.php
Disallow: /boards/viewforum.php
Disallow: /boards/index.php


when migrating urls, but this can help out once it's done. As these will prevent all the advanced pagination sorting options to be cached as well.

So while migrating urls, I'd say, get rid of these for couple month, and put them back after, when bots already stopped using them.

++


In this case its a new forum (not a month old yet), so i dont think google has much indexed yet. Is it better to keep the rules if its a new forum because its not really migrating?
(this is not brandpreventieforum, but the link was removed)
User avatar
emielvenlo
PR0
PR0
 
Posts: 82
Joined: Sun Jan 21, 2007 10:01 am

Postby dcz » Tue Feb 27, 2007 2:17 pm

yes, new install is not like migrating, you can tell bot to ignore these, even though they are redirected ;)

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21376
Joined: Fri Apr 28, 2006 9:03 pm


Return to phpBB2 Forum

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 10 guests