First rules for robots.txt SEO friendly?

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Moderator: Moderators

First rules for robots.txt SEO friendly?

Postby Darth Pincho » Fri Sep 22, 2006 9:03 pm

Hi i wan to know wich are the first things i have to edit in my robots.txt file in order to make my phpbb forum firendly?

I have no clue for that...

I will really appreciate some suggestions.

Thanks
Webmaster at:
http://www.tupincho.net
Discussion board at:
http://www.tupincho.net/foro/
Darth Pincho
PR1
PR1
 
Posts: 146
Joined: Fri Sep 22, 2006 8:37 pm

Advertisement

Postby dcz » Mon Sep 25, 2006 1:21 pm

The robots.txt content depends on you site and your URLs.

As a starting principle you should disallow any page being a duplicate of what you consider as the best URL to offer for each page, obviously if your script outputs more than one kind.

As well, files only meant for example to post messages should be disallowed in order to help bots focusing on your real content, the result of those posts if we follow this example.

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21238
Joined: Fri Apr 28, 2006 9:03 pm

Postby Darth Pincho » Tue Oct 03, 2006 11:39 pm

Here you have the complete robots.txt file at my site

Code: Select all
User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /plugins/
Disallow: /skins/
Disallow: /foro/admin/
Disallow: /foro/db/
Disallow: /foro/files/
Disallow: /foro/images/
Disallow: /foro/includes/
Disallow: /foro/languages/
Disallow: /foro/templates/
Disallow: /foro/posting.php
Disallow: /foro/login.php
Disallow: /foro/memberlist.php
Disallow: /foro/statistics.php
Disallow: /foro/profile.php
Disallow: /foro/viewtopic.php
Disallow: /foro/viewforum.php
Disallow: /foro/index.php?
Disallow: /foro/groupcp.php
Disallow: /foro/search.php
Disallow: /foro/login.php
Disallow: /foro/member
Disallow: /foro/profile.php
Disallow: /foro/faq.php


Any suggestions in order to adsense runs better or other SEO thechniques?
Webmaster at:
http://www.tupincho.net
Discussion board at:
http://www.tupincho.net/foro/
Darth Pincho
PR1
PR1
 
Posts: 146
Joined: Fri Sep 22, 2006 8:37 pm

Postby dcz » Fri Oct 06, 2006 9:39 am

If you had this robots.txt before you found out about this site, then it's very normal you where not spidered, you where disallowing all topics and posts ;)

So keep this robots.txt and add the line suggested in the release topic of the mod rewrite you are using ;)

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21238
Joined: Fri Apr 28, 2006 9:03 pm

Postby Darth Pincho » Fri Oct 06, 2006 9:32 pm

No friend. This is my new robots file after i install mod rewrite mixed.

Because of that I disallow all the php extensions in order to not have duplicates.

In this case is ok? or i have to change something?
Webmaster at:
http://www.tupincho.net
Discussion board at:
http://www.tupincho.net/foro/
Darth Pincho
PR1
PR1
 
Posts: 146
Joined: Fri Sep 22, 2006 8:37 pm

Postby Peter77 » Sat Oct 07, 2006 10:06 pm

Your robots txt is fine. also you may add any other files you do not want search engines to crawl - like any MODs you might have. shop.php, album.php, rss.php ect. 8)
Peter77
phpBB SEO Team
phpBB SEO Team
 
Posts: 524
Joined: Wed May 10, 2006 9:46 am
Location: Michigan

Postby dcz » Sun Oct 08, 2006 10:41 am

I already told you to get rid of this part :

Code: Select all
User-agent: Mediapartners-Google*
Disallow:


and to look at the release topic of the mod rewrite you're using, you miss some of the robots.txt suggested there, such as disallow /post /member ...

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21238
Joined: Fri Apr 28, 2006 9:03 pm


Return to roBots

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 1 guest