| |
| |
|
|
|
|
| |
|
| |
|
| :: |
| Author |
Message |
Peter77
phpBB SEO Team
Joined: 10 May 2006
Posts: 512
Location: Michigan
|
 Posted: Tue Jun 06, 2006 5:15 pm Post subject: About robots.txt |
|
|
Hi, I've been trying for a while to get rid of old URL's of my site from Google. I submited links for spesific subdirectorys to be removed. but google still seems to be searching for these old URL's. or subfolders inside my old phpbb directory too.
In my robots.txt file I have a Dissallow /forum/ the old name of my phpbb. but the directory does not exist on my site... I figured this was okay since search engines usually scan robot.txt first before indexing a site right?
Why would google still be looking for a subdiectory or URL that I have blocked? my robots.txt is under 800 characters so it can't be because I have too many...
should this..
Disallow: /forum/
be
Disallow: /forum
 |
|
|
| Back to top |
|
 |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
| Posted: Tue Jun 06, 2006 6:44 pm Post subject: Re: About robots.txt |
|
|
Well, the problem is, even Google Bots has some trouble sometime following the robots.txt exclusions.
Once it even digged out 8 month old cache version for pages that where last cached like a week before.
And this mean Google do keep a lot of backups 
And after this the GoogleBot started testing old URL, from a previous mod Rewrite that had already been cleared from it's index, again.
So the most secure is to built as well nice redirection, like this :
| Code: | | RewriteRule ^old_folder/ /new_folder/ [R=301,L] |
In your root's .htaccess, before the forum's Rewriterules.
This is pretty basic, www.example.com/old_folder/*.* will be redirected to www.example.com/new_folder/ without the rest of the URI and with a nice 301.
Here I think we should not try to keep the old URI, cause if I remember well, you did not use mod Rewrite before that, so, you don't want old links to your vanilla url (disallowed any way by the robots.txt).
With this code, all the old links will be redirected to your new forum's folder. It's a good target to concentrate PageRank from all those old links, and users will still be "home".
If you prefer to keep the old URI, even though it is a duplicate that you should disallow with your robots.txt, do this instead :
| Code: | | RewriteRule ^old_folder/(.*) /new_folder/$1 [R=301,L] |
But this is more work for the server and you will not transmit the eventual old PageRank to the new URL, because the old one are not mod Rewritten.
You will have things like this :
Byt the way :
is correct |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
Peter77
phpBB SEO Team
Joined: 10 May 2006
Posts: 512
Location: Michigan
|
| Posted: Wed Jun 07, 2006 3:17 pm Post subject: Re: About robots.txt |
|
|
| dcz wrote: | Well, the problem is, even Google Bots has some trouble sometime following the robots.txt exclusions.
Once it even digged out 8 month old cache version for pages that where last cached like a week before.
And this mean Google do keep a lot of backups
And after this the GoogleBot started testing old URL, from a previous mod Rewrite that had already been cleared from it's index, again.
So the most secure is to built as well nice redirection, like this :
| Code: | | RewriteRule ^old_folder/ /new_folder/ [R=301,L] |
In your root's .htaccess, before the forum's Rewriterules.
This is pretty basic, www.example.com/old_folder/*.* will be redirected to www.example.com/new_folder/ without the rest of the URI and with a nice 301.
Here I think we should not try to keep the old URI, cause if I remember well, you did not use mod Rewrite before that, so, you don't want old links to your vanilla url (disallowed any way by the robots.txt).
With this code, all the old links will be redirected to your new forum's folder. It's a good target to concentrate PageRank from all those old links, and users will still be "home".
If you prefer to keep the old URI, even though it is a duplicate that you should disallow with your robots.txt, do this instead :
| Code: | | RewriteRule ^old_folder/(.*) /new_folder/$1 [R=301,L] |
But this is more work for the server and you will not transmit the eventual old PageRank to the new URL, because the old one are not mod Rewritten.
You will have things like this :
Byt the way :
is correct |
Thanks for the rewriterule, it works like a charm! lol, yeah it's amazing what some search engines still keep in thier archives. latley ive been getting hit for searches of Music from way back before I even installed phpbb and I think this rewriterule you just presented will be very handy. |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
|
| Back to top |
|
|
|
| Navigation |
Similar Topics |
|
|
|
|
|
|
|
| |
|
|
|
|
| |
|
|
|
|
| |
