roBots : My robots

My robots

phpBB SEO » SEO Forum » roBots

Author

Message

fabiomaticus

Joined: 06 Jun 2007
Posts: 42
Location: Taranto - Italy

Posted: Tue Jun 19, 2007 8:50 pm Post subject: My robots

Please this is my robots.txt

Code:

User-agent: *
Disallow: forum/post-*.html$
Disallow: forum/updates-topic.html*$
Disallow: forum/stop-updates-topic.html*$
Disallow: forum/ptopic*.html$
Disallow: forum/ntopic*.html$
Disallow: forum/viewtopic.php
Disallow: forum/viewforum.php
Disallow: forum/login.php
Disallow: forum/profile.php
Disallow: forum/index.php?
Disallow: forum/posting.php
Disallow: forum/groupcp.php
Disallow: forum/search.php
Disallow: forum/privmsg.php
Disallow: forum/post
Disallow: forum/member
Disallow: forum/memberlist.php
Disallow: forum/faq.php

Is it ok?

I have this mod installed:
- Guest Session MOD.
- Search Engime Optimization.
- SEO Advanced mod Rewrite.
- SEO Dynamic Metatags.
- Static URLs mod Rewrite.
- Mx Ggsitemaps.
- Mx Sitemaps Auto Link Add on phpBB.
- Mx Sitemaps.
- Google Yahoo MSN Sitemaps and RSS

AmirAbbas
phpBB SEO Team

Joined: 11 May 2006
Posts: 529
Location: IRAN

Posted: Wed Jun 20, 2007 6:13 am Post subject: Re: My robots

hi fabiomaticus
welcome to phpbb-seo.com

first tell us that what the path of your forum.
have you installed it in sub domain ?
i saw that you are using wildcard in you robots.txt file. wildcard is not standard character for robots.txt

instead of this

Code:

Disallow: forum/post-*.html$
Disallow: forum/updates-topic.html*$
Disallow: forum/stop-updates-topic.html*$
Disallow: forum/ptopic*.html$
Disallow: forum/ntopic*.html$

you can use this one

Code:

Disallow: forum/post
Disallow: forum/updates-topic.html
Disallow: forum/stop-updates-topic.html
Disallow: forum/ptopic
Disallow: forum/ntopic

i remember that this robots.txt rules are belong to able2know mod. if you are using advance rewrite mod. you don't need them.
just add adcance zero duplicate mod to your installed mods. this mod is very useful and it can redirect post URLs to proper topic URLs.

another thing, i saw you have installed Static URLs mod Rewrite
what is this ? if you have installed advance rewrite mod you don't need any other rewrite mod

_________________
طراحی وب ، آموزش سئو ، آموزش CSS ، phpbb فارسی ، phpbb3 فارسی ، دروپال فارسی
طراحی وب ، قالب وبلاگ ، استاندارد وب ، آموزش CSS و HTML ، آموزش php

fabiomaticus

Joined: 06 Jun 2007
Posts: 42
Location: Taranto - Italy

Posted: Wed Jun 20, 2007 2:50 pm Post subject: Re: My robots

My site is www.500forum.it and my phpbb forum is in forum folder
-www.500forum.it/forum

Now my robots.txt is

Code:

User-agent: *
Disallow: forum/post
Disallow: forum/updates-topic.html
Disallow: forum/stop-updates-topic.html
Disallow: forum/ptopic
Disallow: forum/ntopic
Disallow: forum/viewtopic.php
Disallow: forum/viewforum.php
Disallow: forum/login.php
Disallow: forum/profile.php
Disallow: forum/index.php?
Disallow: forum/posting.php
Disallow: forum/groupcp.php
Disallow: forum/search.php
Disallow: forum/privmsg.php
Disallow: forum/post
Disallow: forum/member
Disallow: forum/memberlist.php
Disallow: forum/faq.php

I have this mod installed:
- Guest Session MOD.
- Search Engime Optimization.
- SEO Advanced mod Rewrite.
- SEO Dynamic Metatags.
- Mx Ggsitemaps.
- Mx Sitemaps Auto Link Add on phpBB.
- Mx Sitemaps.
- Google Yahoo MSN Sitemaps and RSS
- adcance zero duplicate mod

fabiomaticus

Joined: 06 Jun 2007
Posts: 42
Location: Taranto - Italy

Posted: Wed Jun 20, 2007 2:51 pm Post subject: Re: My robots

This is my .htaccess

Code:

# Options +FollowSymlinks
RewriteEngine On
RewriteBase /

#########################################################
# PHPBB SEO REWRITE RULES          #
#########################################################
# AUTHOR : DCZ http://www.phpbb-seo.com/
# STARTED : 01/2006
#####################################################
# FORUMS PAGES
########################
# FORUM PROTECTION RULE
RewriteRule ^forum/.+/([^/]+\.html)$ /forum/index.php [R=301,L]
# CATEGORIES
RewriteRule ^forum/.+-vc([0-9]+)\.html$ /forum/index.php?c=$1 [QSA,L]
# PAGINATED FORUM
RewriteRule ^forum/.+-vf([0-9]+)-([0-9]+)\.html$ /forum/viewforum.php?f=$1&start=$2 [QSA,L]
# FORUM
RewriteRule ^forum/.+-vf([0-9]+)\.html$ /forum/viewforum.php?f=$1 [QSA,L]
# PAGINATED TOPIC
RewriteRule ^forum/.+-vt([0-9]+)-([0-9]+)\.html$ /forum/viewtopic.php?t=$1&start=$2 [QSA,L]
# TOPIC
RewriteRule ^forum/.+-vt([0-9]+)\.html$ /forum/viewtopic.php?t=$1 [QSA,L]
# POST
RewriteRule ^forum/post([0-9]+)\.html$ /forum/viewtopic.php?p=$1 [QSA,L]
#PROFILES
RewriteRule ^forum/member([0-9]+)\.html$ /forum/profile.php?mode=viewprofile&u=$1 [QSA,L]
# END PHPBB PAGES
#####################################################RewriteEngine On
RewriteRule ^forum/index\.html$ /forum/index.php [QSA,L]

RewriteRule ^sitemaps([0-9]+)\.html$ /forum/sitemaps.php?c=$1 [QSA,L]
RewriteRule ^forum/sitemaps.html$ /forum/sitemaps.php [QSA,L]
RewriteRule ^sitemaps\.xml$ /forum/sitemap.php [L]

RewriteRule ^forum-map\.html$ /forum/sitemaps.php?fim [QSA,L]

RewriteRule ^forum-m\ap\.html$ /sitemaps.php?fim [QSA,L]
RewriteRule ^forum-m\ap([0-9]+)-([0-9]+)\.html$ /sitemaps.php?fmp=$1&start=$2 [QSA,L]
RewriteRule ^forum-m\ap([0-9]+)\.html$ /sitemaps.php?fmp=$1 [QSA,L]

RewriteRule ^.+-fmp([0-9]+)-([0-9]+)\.html$ /forum/sitemaps.php?fmp=$1&start=$2 [QSA,L]
RewriteRule ^.+-fmp([0-9]+)\.html$ /forum/sitemaps.php?fmp=$1 [QSA,L]
RewriteRule ^.+-sc([0-9]+)\.html$ /forum/sitemaps.php?c=$1 [QSA,L]

RewriteRule ^forum-sitemap-([0-9]+)\.xml$ /forum/sitemap.php?fid=$1 [L]
RewriteRule ^forum/forum-sitemap-([0-9]+).xml$ /forum/vsitemap.php?fid=$1 [L]
RewriteRule ^forum/sitemaps.xml$ /forum/sitemap.php [L]

RewriteRule ^sitemap-forum\.xml$ /forum/sitemap.php?forum [L]
RewriteRule ^forum/sitemap-forum.xml$ /forum/sitemap.php?forum [L]

#########################################################
# GYM SITEMAPS AND RSS REWRITE RULES       #
#########################################################
# AUTHOR : dcz http://www.phpbb-seo.com/
# STARTED : 2006/02/22
########################
# RSS main
RewriteRule ^forum/rss-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /forum/rss.php?$1&$2 [L]
# RSS forums
RewriteRule ^forum/forums-rss-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /forum/rss.php?forum&c&$1&$2 [L]
# RSS all
RewriteRule ^forum/([a-zA-Z0-9_-]+)-rss([0-9]*)-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /forum/rss.php?$1=$2&$3&$4 [L]
# RSS forum topics
RewriteRule ^forum/.+-rf([0-9]+)-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /forum/rss.php?forum=$1&$2&$3 [L]
# SitemapIndex
RewriteRule ^forum/sitemaps\.(xml(\.gz)?)$ /forum/sitemap.php [L]
# Sitemap modules
RewriteRule ^forum/([a-zA-Z0-9_-]+)-sitemap\.(xml(\.gz)?)$ /forum/sitemap.php?$1 [L]
# Forum Sitemaps
RewriteRule ^forum/.+-gf([0-9]+)\.(xml(\.gz)?)$ /forum/sitemap.php?forum=$1 [L]
# Yahoo! urllist.txt
RewriteRule ^forum/urllist\.(txt(\.gz)?)$ /forum/urllist.php [L]
#########################################################
# END GYM SITEMAPS AND RSS REWRITE RULES    #
#########################################################

AmirAbbas
phpBB SEO Team

Joined: 11 May 2006
Posts: 529
Location: IRAN

Posted: Thu Jun 21, 2007 10:23 am Post subject: Re: My robots

everything is ok
your robots.txt file is in correct place.
zero duplicate mod working corrcetly and all redirection is http 301 redirect that is completly safe.

just a few questions.
you have these rules in your robots.txt file

Code:

Disallow: forum/updates-topic.html
Disallow: forum/stop-updates-topic.html
Disallow: forum/ptopic
Disallow: forum/ntopic

this robots.txt rules are belongs to able2know mod. if you were using able2know mod before you must keep them but if you have installed mod rewrite for the first time you can remove that rules form your robots.txt file.

now you have installed zero duplicate. you can remove this lines form your robots.txt file

Quote:

Disallow: forum/post
Disallow: forum/viewtopic.php
Disallow: forum/viewforum.php

because zero duplicate can handle these URLs. for example in your forum there is some post urls like this one

-http://www.500forum.it/forum/post6767.html#6767

click on the above link and see the result in address bar of your browser. as you can see the post URL will redirect to proper topic URL. due to the presence of zero duplicate mod you can remove the rules in your robots.txt file that i mentioned above

i think you have installed simple subforum mod. you need simple subforum patch too. because without this patch of the urls are in old shape (with .php prefix). notice that zero duplicate can redirect that old shape urls too but its better for you to install the simple subforum patch

you wrote that you have installed Mx Sitemaps Auto Link Add on phpBB
i can not see the link that provided by autolink at the buttom of your pages.

at last
dear fabiomaticus
i can not see phpbb-seo.com link at the buttom of your page. please read this topic

Wink

fabiomaticus

Joined: 06 Jun 2007
Posts: 42
Location: Taranto - Italy

Posted: Thu Jun 21, 2007 11:40 am Post subject: Re: My robots

AmirAbbas wrote:

just a few questions.
you have these rules in your robots.txt file

Code:

Disallow: forum/updates-topic.html
Disallow: forum/stop-updates-topic.html
Disallow: forum/ptopic
Disallow: forum/ntopic

Quote:

Disallow: forum/post
Disallow: forum/viewtopic.php
Disallow: forum/viewforum.php

Ok, I have modified the file robots.txt; now it is

Code:

User-agent: *
Disallow: forum/login.php
Disallow: forum/profile.php
Disallow: forum/posting.php
Disallow: forum/groupcp.php
Disallow: forum/search.php
Disallow: forum/privmsg.php
Disallow: forum/post
Disallow: forum/member
Disallow: forum/memberlist.php
Disallow: forum/faq.php

Quote:

i think you have installed simple subforum mod. you need simple subforum patch too. because without this patch of the urls are in old shape (with .php prefix). notice that zero duplicate can redirect that old shape urls too but its better for you to install the simple subforum patch

Ok, I have now installed this mod!

Quote:

you wrote that you have installed Mx Sitemaps Auto Link Add on phpBB
i can not see the link that provided by autolink at the buttom of your pages.

Now, there are!

AmirAbbas
phpBB SEO Team

Joined: 11 May 2006
Posts: 529
Location: IRAN

Posted: Fri Jun 22, 2007 5:28 pm Post subject: Re: My robots

you still have this rule in your robots.txt file

Code:

Disallow: forum/post

this line was duplicted in your robots.txt file. first remove it
after that you must go to

phpbb control panel > phpbb SEO
on that section you must have three part based on the mod that you have installed:

GYM SiteMaps & RSS
SiteMaps
SitemapIndex

first, go to GYM SiteMaps & RSS and find URL Rewriting and turn it on. below that part you have URL rewriting type that you must set it to advance. after that click on submit button of that section

second, go to SiteMaps and find Mod rewrite for Sitemaps and turn it on

just it Wink

fabiomaticus

Joined: 06 Jun 2007
Posts: 42
Location: Taranto - Italy

Posted: Sat Jun 23, 2007 8:47 pm Post subject: Re: My robots

AmirAbbas wrote:

you still have this rule in your robots.txt file

Code:

Disallow: forum/post

this line was duplicted in your robots.txt file. first remove it

Ok, now my robots is

Code:

User-agent: *
Disallow: forum/login.php
Disallow: forum/profile.php
Disallow: forum/posting.php
Disallow: forum/groupcp.php
Disallow: forum/search.php
Disallow: forum/privmsg.php
Disallow: forum/member
Disallow: forum/memberlist.php
Disallow: forum/faq.php

Quote:

first, go to GYM SiteMaps & RSS and find URL Rewriting and turn it on. below that part you have URL rewriting type that you must set it to advance. after that click on submit button of that section

second, go to SiteMaps and find Mod rewrite for Sitemaps and turn it on

just it Wink

Ok!!!

fabiomaticus

Joined: 06 Jun 2007
Posts: 42
Location: Taranto - Italy

Posted: Mon Jul 09, 2007 12:39 pm Post subject: Re: My robots

Please, i have the forum installed on my root.
Is it correct my .htaccess

Code:

# Options +FollowSymlinks
RewriteEngine On
RewriteEngine On
RewriteBase /

#########################################################
# PHPBB SEO REWRITE RULES #
#########################################################
# AUTHOR : DCZ http://www.phpbb-seo.com/
# STARTED : 01/2006
#####################################################
# FORUMS PAGES
########################
# FORUM PROTECTION RULE
# RewriteRule ^.+/([^/]+\.html)$ /index.php [R=301,L]
# CATEGORIES
# RewriteRule ^.+-vc([0-9]+)\.html$ /index.php?c=$1 [QSA,L]
# PAGINATED FORUM
RewriteRule ^.+-vf([0-9]+)-([0-9]+)\.html$ /viewforum.php?f=$1&start=$2 [QSA,L]
# FORUM
RewriteRule ^.+-vf([0-9]+)\.html$ /viewforum.php?f=$1 [QSA,L]
# PAGINATED TOPIC
RewriteRule ^.+-vt([0-9]+)-([0-9]+)\.html$ /viewtopic.php?t=$1&start=$2 [QSA,L]
# TOPIC
RewriteRule ^.+-vt([0-9]+)\.html$ /viewtopic.php?t=$1 [QSA,L]
# POST
RewriteRule ^post([0-9]+)\.html$ /viewtopic.php?p=$1 [QSA,L]
#PROFILES
RewriteRule ^member([0-9]+)\.html$ /profile.php?mode=viewprofile&u=$1 [QSA,L]
# END PHPBB PAGES
#####################################################

RewriteEngine On
# RewriteRule ^index\.html$ /index.php [QSA,L]

RewriteRule ^sitemaps([0-9]+)\.html$ /sitemaps.php?c=$1 [QSA,L]
RewriteRule ^sitemaps.html$ /sitemaps.php [QSA,L]
RewriteRule ^sitemaps\.xml$ /sitemap.php [L]

RewriteRule ^forum-map\.html$ /sitemaps.php?fim [QSA,L]

RewriteRule ^forum-m\ap\.html$ /sitemaps.php?fim [QSA,L]
RewriteRule ^forum-m\ap([0-9]+)-([0-9]+)\.html$ /sitemaps.php?fmp=$1&start=$2 [QSA,L]
RewriteRule ^forum-m\ap([0-9]+)\.html$ /sitemaps.php?fmp=$1 [QSA,L]

RewriteRule ^.+-fmp([0-9]+)-([0-9]+)\.html$ /sitemaps.php?fmp=$1&start=$2 [QSA,L]
RewriteRule ^.+-fmp([0-9]+)\.html$ /sitemaps.php?fmp=$1 [QSA,L]
RewriteRule ^.+-sc([0-9]+)\.html$ /sitemaps.php?c=$1 [QSA,L]

RewriteRule ^forum-sitemap-([0-9]+)\.xml$ /sitemap.php?fid=$1 [L]
RewriteRule ^forum-sitemap-([0-9]+).xml$ /vsitemap.php?fid=$1 [L]
RewriteRule ^sitemaps.xml$ /forum/sitemap.php [L]

RewriteRule ^sitemap-forum\.xml$ /sitemap.php?forum [L]
RewriteRule ^sitemap-forum.xml$ /sitemap.php?forum [L]

#########################################################
# GYM SITEMAPS AND RSS REWRITE RULES #
#########################################################
# AUTHOR : dcz http://www.phpbb-seo.com/
# STARTED : 2006/02/22
########################
# RSS main
RewriteRule ^rss-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?$1&$2 [L]
# RSS forums
RewriteRule ^forums-rss-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?forum&c&$1&$2 [L]
# RSS all
RewriteRule ^([a-zA-Z0-9_-]+)-rss([0-9]*)-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?$1=$2&$3&$4 [L]
# RSS forum topics
RewriteRule ^.+-rf([0-9]+)-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?forum=$1&$2&$3 [L]
# SitemapIndex
RewriteRule ^sitemaps\.(xml(\.gz)?)$ /sitemap.php [L]
# Sitemap modules
RewriteRule ^([a-zA-Z0-9_-]+)-sitemap\.(xml(\.gz)?)$ /sitemap.php?$1 [L]
# Forum Sitemaps
RewriteRule ^.+-gf([0-9]+)\.(xml(\.gz)?)$ /sitemap.php?forum=$1 [L]
# Yahoo! urllist.txt
RewriteRule ^urllist\.(txt(\.gz)?)$ /urllist.php [L]
#########################################################
# END GYM SITEMAPS AND RSS REWRITE RULES #
#########################################################

#########################################################
# SMARTOR ALBUM REWRITE RULES FAP #
#########################################################
# AUTHOR : dcz http://www.phpbb-seo.com/
# STARTED : 2007/02/22
########################
# ALBUM INDEX
RewriteRule ^album\.html$ /album.php [QSA,L,NC]
# ALBUM PAGINATED CAT
RewriteRule ^[a-z0-9_-]+-ac([0-9]+)-([0-9]+)\.html$ /album_cat.php?cat_id=$1&start=$2 [QSA,L,NC]
# ALBUM CAT
RewriteRule ^[a-z0-9_-]+-ac([0-9]+)\.html$ /album_cat.php?cat_id=$1 [QSA,L,NC]
# ALBUM PAGINATED PIC
RewriteRule ^[a-z0-9_-]+-sp([0-9]+)-([0-9]+)\.html$ /album_showpage.php?pic_id=$1&start=$2 [QSA,L,NC]
# ALBUM PIC with full option
RewriteRule ^[a-z0-9_-]+-sp([0-9]+)-?(full)?\.html$ /album_showpage.php?pic_id=$1&$2 [QSA,L,NC]
# PERSONAL USER GALLERY PAGINATED
RewriteRule ^[a-z0-9_-]+-ap([0-9]+)-([0-9]+)-?(list|all)?\.html$ /album.php?user_id=$1&start=$2&mode=$3 [QSA,L,NC]
# PERSONAL USER GALLERY
RewriteRule ^[a-z0-9_-]+-ap([0-9]+)-?(list|all)?\.html$ /album.php?user_id=$1&mode=$2 [QSA,L,NC]
# PERSONAL USER CAT PAGINATED
RewriteRule ^[a-z0-9_-]+-cp([0-9]+)-([0-9]+)-([0-9]+)\.html$ /album_cat.php?user_id=$1&cat_id=$2&start=$3 [QSA,L,NC]
# PERSONAL USER CAT
RewriteRule ^[a-z0-9_-]+-cp([0-9]+)-([0-9]+)\.html$ /album_cat.php?user_id=$1&cat_id=$2 [QSA,L,NC]
# PERSONAL CAT PAGINATED
RewriteRule ^member-galleries-([0-9]+)\.html$ /album_personal_index.php?start=$1 [QSA,L,NC]
# PERSONAL CAT
RewriteRule ^member-galleries\.html$ /album_personal_index.php [QSA,L,NC]
# ALL PICS PAGINATED
RewriteRule ^all-pics-([0-9]+)\.html$ /album_allpics.php?start=$1 [QSA,L,NC]
# ALL PICS
RewriteRule ^all-pics\.html$ /album_allpics.php [QSA,L,NC]
# THUMBNAILS
RewriteRule ^[a-z0-9_-]+-thumb([0-9]+)\.jpg$ /album_thumbnail.php?pic_id=$1 [QSA,L,NC]
# MED
RewriteRule ^[a-z0-9_-]+-mid([0-9]+)\.jpg$ /album_picm.php?pic_id=$1 [QSA,L,NC]
# FULL
RewriteRule ^[a-z0-9_-]+-pic([0-9]+)\.jpg$ /album_pic.php?pic_id=$1 [QSA,L,NC]
#########################################################

dcz
Administrateur - Site Admin

Joined: 28 Apr 2006
Posts: 13354

Posted: Wed Jul 11, 2007 7:26 pm Post subject: Re: My robots

If all links works, then it's ok, but it's really off topic to talk about .htaccess in this topic. ++
_________________ Useful links : SEO Forum \|\| SEO Directory \|\| SEO phpBB \|\| SEO phpBB3 \|\| Search ____________________ Liens Utiles : Forum référencement \|\| Annuaire référencement \|\| Référencement phpBB \|\| Référencement phpBB3 \|\| Recherche

Display posts from previous:

	phpBB SEO » SEO Forum » roBots
Page 1 of 1

Navigation