Spiders leading to 404 Pages

phpBB2 SEO Advanced mod Rewrite support forum.
This mods performs URL rewriting for phpBB, injecting categories, forums and topic titles in their URLs.

Moderator: Moderators

Spiders leading to 404 Pages

Postby altimaforums » Wed Jun 27, 2007 7:51 pm

I've tested it on Google Webmaster Tools also.

http://www.webconfs.com/search-engine-s ... ulator.php

Basically, Instead of..

http://www.website.com/forums/thefile

The spider is going in to http://www.altimaforums.net and getting 301 redirected to http://www.altimaforums.net/forums

.. but the spider still thinks the files are at / (root).

Code: Select all
# You could need to un-comment the following line
# Options +FollowSymlinks
RewriteEngine On
RewriteBase /

redirect 301 /index.php http://www.altimaforums.net/forums/index.html

#########################################################
# PHPBB SEO REWRITE RULES            #
#########################################################
# AUTHOR : DCZ http://www.phpbb-seo.com/
# STARTED : 01/2006
#####################################################
# FORUMS PAGES
########################
# FORUM INDEX (un-comment if used)
RewriteRule ^forums/index\.html$ /forums/index.php [QSA,L,NC]
# FORUM PROTECTION RULE
# RewriteRule ^forums/[a-z0-9_-]*/([^/]+\.html)$ /forums/index.php [R=301,L,NC]
# CATEGORIES
RewriteRule ^forums/[a-z0-9_-]*-c([0-9]+)\.html$ /forums/index.php?c=$1 [QSA,L,NC]
# PAGINATED FORUM
RewriteRule ^forums/[a-z0-9_-]*-f([0-9]+)-([0-9]+)\.html$ /forums/viewforum.php?f=$1&start=$2 [QSA,L,NC]
# FORUM
RewriteRule ^forums/[a-z0-9_-]*-f([0-9]+)\.html$ /forums/viewforum.php?f=$1 [QSA,L,NC]
# PAGINATED TOPIC
RewriteRule ^forums/[a-z0-9_-]*-t([0-9]+)-([0-9]+)\.html$ /forums/viewtopic.php?t=$1&start=$2 [QSA,L,NC]
# TOPIC
RewriteRule ^forums/[a-z0-9_-]*-t([0-9]+)\.html$ /forums/viewtopic.php?t=$1 [QSA,L,NC]
# POST
RewriteRule ^forums/post([0-9]+)\.html$ /forums/viewtopic.php?p=$1 [QSA,L,NC]
# PROFILES
RewriteRule ^forums/member([0-9]+)\.html$ /forums/profile.php?mode=viewprofile&u=$1 [QSA,L,NC]
# FIX WWW
RewriteCond %{HTTP_HOST} !^www\.altimaforums\.net$ [NC]
RewriteRule ^(.*) http://www.altimaforums.net/$1 [QSA,L,R=301]
# END PHPBB PAGES
#########################################################

#########################################################
# GYM SITEMAPS AND RSS REWRITE RULES         #
#########################################################
# AUTHOR : dcz http://www.phpbb-seo.com/
# STARTED : 2006/02/22
########################
# RSS main
RewriteRule ^rss-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?$1&$2 [L]
# RSS forums
RewriteRule ^forums-rss-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?forum&c&$1&$2 [L]
# RSS all
RewriteRule ^([a-zA-Z0-9_-]+)-rss([0-9]*)-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?$1=$2&$3&$4 [L]
# RSS forum topics
RewriteRule ^.+-rf([0-9]+)-?(l|s)?-?(m)?\.(xml(\.gz)?)$ /rss.php?forum=$1&$2&$3 [L]
# SitemapIndex
RewriteRule ^sitemaps\.(xml(\.gz)?)$ /sitemap.php [L]
# Sitemap modules
RewriteRule ^([a-zA-Z0-9_-]+)-sitemap\.(xml(\.gz)?)$ /sitemap.php?$1 [L]
# Forum Sitemaps
RewriteRule ^.+-gf([0-9]+)\.(xml(\.gz)?)$ /sitemap.php?forum=$1 [L]
# Yahoo! urllist.txt
RewriteRule ^urllist\.(txt(\.gz)?)$ /urllist.php [L]
#########################################################
# END GYM SITEMAPS AND RSS REWRITE RULES      #
#########################################################


Anybody know why this is?

Thanks
altimaforums
 
Posts: 4
Joined: Mon Jun 04, 2007 2:46 am

Advertisement

Postby SeO » Wed Jun 27, 2007 8:29 pm

Looks like it is the part you added :
Code: Select all
redirect 301 /index.php http://www.altimaforums.net/forums/index.html


The purpose seems to be the one you describe as problematic.

I don't think bot would have any problem with this, they are just redirected from root to the forum index with a genuine HTTP 301 redirection. Now if the spider simulation tool your tried was not able to deal with it is another problem, spider bot do follow HTTP 301.

As well, it's not necessarily the best thing to do to HTTP 301 redirect the main domain's root to a sub-folder, a simple welcome page could start building PageRank, good to be transmitted in the deeper folders, while here you only build some for the forum index.
SeO
Admin
Admin
 
Posts: 6333
Joined: Wed Mar 15, 2006 9:41 pm

Postby altimaforums » Wed Jun 27, 2007 8:36 pm

Thanks for the reply SEO.

This is what its doing (via spider sim.)

Spidered Links :
Code: Select all
http://www.altimaforums.net/forums
http://www.altimaforums.net/profile.php?mode=register
http://www.altimaforums.net/index.php
http://www.altimaforums.net/forums/
http://www.altimaforums.net/forums/
http://www.altimaforums.net/memberlist.php
http://www.altimaforums.net/search.php
http://www.altimaforums.net/donate.php
http://www.altimaforums.net/garage.php
http://www.altimaforums.net/profile.php?mode=register
http://www.altimaforums.net/profile.php?mode=editprofile
http://www.altimaforums.net/login.php?redirect=index.php
http://www.altimaforums.net/the-first-generation-f5.html
http://www.altimaforums.net/images/lightbox/5-1.jpg
http://www.altimaforums.net/the-second-generation-f6.html
http://www.altimaforums.net/images/lightbox/6-1.jpg
http://www.altimaforums.net/the-third-generation-f7.html
http://www.altimaforums.net/images/lightbox/7-1.jpg
http://www.altimaforums.net/the-fourth-generation-f8.html
http://www.altimaforums.net/images/lightbox/8-1.jpg
http://www.altimaforums.net/post101.html#101
http://www.altimaforums.net/member6.html
http://www.altimaforums.net/post101.html#101
http://www.altimaforums.net/viewtopic.php?t=35
http://www.altimaforums.net/profile.php?mode=viewprofile$amp;u=6


etc
altimaforums
 
Posts: 4
Joined: Mon Jun 04, 2007 2:46 am

Postby dcz » Thu Jun 28, 2007 8:00 am

Well, the spider tool you are using really seems to be causing this by itself, it must be simulating bots from the 80's or something like this :mrgreen: .

Google does not have any problem to spider you board with the new rewritten urls : http://www.google.fr/search?q=site%3Aww ... Fforums%2F

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21219
Joined: Fri Apr 28, 2006 9:03 pm

Postby altimaforums » Thu Jun 28, 2007 5:47 pm

haha @ 80's.

Okay, thanks! I appreciate it much.
altimaforums
 
Posts: 4
Joined: Mon Jun 04, 2007 2:46 am


Return to phpBB2 Advanced mod Rewrite

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 3 guests