Hi all. I recently did a SEO audit of my forums at http://www.bearsfansonline.com and it showed that I have over 18,000 duplicate pages. When I went and clicked on some of the links it showed I saw that a lot of them were "reply" links. By this I mean it was the page after you click "reply" to a post.

Apparently when the site crawled my website it followed the "reply" links for all the topics. Because it wasn't registered, it just went to the Login page. However, each link was unique so when it got the login page for all of them it assumed they were duplicates. Here's an example of what I mean:

-http://www.bearsfansonline.com/forum/posting.php?mode=reply&f=4&t=4272

-http://www.bearsfansonline.com/forum/posting.php?mode=reply&f=6&t=5051

-http://www.bearsfansonline.com/forum/posting.php?mode=reply&f=4&t=18

While all different, they lead to the same page. Will the mod correct this problem?

Groups’ forum permissions for bots shall be "Bot Access" for all forums you want to be crawled / indexed. You can put /posting.php in your robots.txt or make it "noindex, follow" (either with the 'File filter" feature of phpBB-SEO or by using the X-Robots Tag function of Apache if mod_headers.c is loaded).

.htaccess:

Code: Select all

<IfModule mod_headers.c>
<FilesMatch "posting\.php">
Header set X-Robots-Tag "noindex, follow"
</FilesMatch>
</IfModule>


Ok, I made sure bots had "Bot Access" for all the relevant forums. Do I just copy and paste that code into the .htaccess file or do I need to verify that mod_headers.c is loaded? Here's the current content of my .htaccess:

Code: Select all

#
# Uncomment the statement below if you want to make use of
# HTTP authentication and it does not already work.
# This could be required if you are for example using PHP via Apache CGI.
#
#<IfModule mod_rewrite.c>
#RewriteEngine on
#RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
#</IfModule>

<Files "config.php">
Order Allow,Deny
Deny from All
</Files>

<Files "common.php">
Order Allow,Deny
Deny from All
</Files>


Thanks a lot for your help.

I'm confused. Do you have any phpBB SEO mods installed?

Xee wrote:While all different, they lead to the same page. Will the mod correct this problem?

Yes, it will correct that problem, though you should practice on a dev site and not a live installation, unless you don't mind quickly restoring your site from the backups you created before you tried to install it if you flub a code edit.

You also need to install no dupe and zero dupe. You have a different style than prosilver or subsilver2, so make sure you also make the template changes appropriate to that style.

coyote wrote:I'm confused. Do you have any phpBB SEO mods installed?

None whatsoever. I was on the fence about installing the Premod but don't have the time right now to fix any hiccups if it doesn't play nice with my existing mods. That's why I was wondering if I could get away with just installing the Ultimate mod.

So, will that .htaccess edit only work if I have the mod installed?

It will work fine, but it won't solve your problem. The only thing it does as written now is provide extra security for your config.php and common.php files.

How would Stephen26's suggested edit increase security? It seems as though the code he suggested I add would stop the bots from following the "Post Reply" button/link? Or am I reading it wrong?

Xee wrote:Ok, I made sure bots had "Bot Access" for all the relevant forums. Do I just copy and paste that code into the .htaccess file or do I need to verify that mod_headers.c is loaded?

Search engine bots don't 'see' the "post reply" / "new topic" buttons anymore on your forum now that they just have bot access. You can verify that with User Agent Switcher if you want (I already did).

The code I gave you does check if mod_headers.c is loaded, but just adding /posting.php to your robots.txt is fine too.

Stephen26 wrote:

Xee wrote:Ok, I made sure bots had "Bot Access" for all the relevant forums. Do I just copy and paste that code into the .htaccess file or do I need to verify that mod_headers.c is loaded?

Search engine bots don't 'see' the "post reply" / "new topic" buttons anymore on your forum now that they just have bot access. You can verify that with User Agent Switcher if you want (I already did).

The code I gave you does check if mod_headers.c is loaded, but just adding /posting.php to your robots.txt is fine too.

Hmm, I may stand corrected. Stephen is right. Just making sure bots have "bot access" actually probably does go a long way towards solving it.

Nice site, btw.

Xee wrote:How would Stephen26's suggested edit increase security? It seems as though the code he suggested I add would stop the bots from following the "Post Reply" button/link? Or am I reading it wrong?

no, you're reading it right, I was just fixated on how you could install the mod lol. Those lines in the .htaccess should be in everyone's phpBB installation whether they have phpBB SEO or not. Not the lines that are commented (the ones that begin with #), but the ones that deny access to everyone from calling config.php or common.php from a browser (or another server) and gaining access to your database. But they have nothing to do with what bots see or not.

this

Code: Select all

<Files "config.php">
Order Allow,Deny
Deny from All
</Files>

<Files "common.php">
Order Allow,Deny
Deny from All
</Files>

OK, some clarification. robots.txt is a file you place at the root of your server to prevent search engines from following certain links. Only search engines look for it or even know it's there (though you can call any robots.txt for any site through a browser. Try it. a lot of sites have them. -www.anysite.com/robots.txt). .htaccess is a file that only your server can read and it affects everyone. It makes no distinction between a bot, a user, the owner of the site, or a remote server. .htaccess also only affects incoming requests, and cannot change outputted links. It can never be called in a browser or read by a 3rd party. If that helps.