| :: |
| Author |
Message |
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
 Posted: Sat May 27, 2006 7:19 am Post subject: how can i remove my old URLs from google database |
|
|
hello
i want to ask a question
i have forum form 2 years ago
at first i had installed able2knoe rewrite mod
after six month i remove this mod and i used standard phpbb urls
at this time i have installed phpbb seo mod
i have 2 different url for each topic (and in near future a will have three diffirent url after installing phpbb seo mod)
i heard that there is service in google that you can remove all of old url form google database
how can i remove these old urls and let the google to index my page again
thanks
 |
|
|
| Back to top |
|
 |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
|
| Back to top |
|
|
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
| Posted: Mon May 29, 2006 5:20 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
i have some problem
at first is about my domain
at this time i have installed phpbb in subdomain
in this address
http://forum.amuzesh.org
yesterday i used site:forum.amuzesh.org in google
i have over 5000 record in google database
first
over 80 percent of this urls are broken because i had used able2know rewritemod
last year and i removed it 2 month ago
there is near 19 percent url with standard phpbb url
another big problem
some of the link in google database is something like this
http://forum.amuzesh.org/viewforum.php?f=7
and i have another url for same topic with this address in googlw database
http://www.amuzesh.org/forum/viewforum.php?f=7
i think it can make problem
and with form is better
first form or second form
you said that i must make a robots.txt file and i must ban the standard urls
and able2know urls but at this time all of this 5000 URLs are useless
over 80 percent of them are broken and other 20 percent is in standard form
what can i do ? |
_________________
چهار گوش - طراحی وب - مجله طراحی وب |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
| Posted: Mon May 29, 2006 5:42 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
First apply the required code change to force to use or not to use the www prefix.
Since http://www.forum.amuzesh.org/ works, I'd suggest to keep it like this, with the www, but it's as you wish, the difference is minor, all of this is commented in the www prefix topic
This way, anything that's not your real URL (thus the sub domain) will be redirected with a nice http 301 to the real one, keeping the URI.
So that's why you will still need to disallow all the unwanted URL in your robots.txt file.
This should do it :
| Code: | User-agent: *
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /search.php
Disallow: /login.php
Disallow: /faq.php
Disallow: /about |
Noticed "Disallow: /about" ? you should as well disallow in the same way any other unused able2know URL (pagination may be).
| amir abbas wrote: |
i think it can make problem
and with form is better
first form or second form
you said that i must make a robots.txt file and i must ban the standard urls
and able2know urls but at this time all of this 5000 URLs are useless
over 80 percent of them are broken and other 20 percent is in standard form
what can i do ? |
This is always a problem to change URL. Even more with a lot of them.
The www prefix fix will transmit your Back-links and thus your PageRank to your sub domain, but your topics will have to be re indexed.
We can try and see if a rewriterule would no do the trick, I have to check.
Anyway, since this is already done and since bots must have already encountered a lot of 404's on the old url, you should get rid of them using the automatic URL removal system on this page http://www.google.com/support/webmasters/bin/answer.py?answer=34440&topic=8459
And keep the robots.txt.
Wait a bit though to implement the disallow on able2know URL, I have to check if a redirection would be possible, you should thus first take care of the www prefix, then disallow all suggested URL (without the last line with about) and tell Google to get rid of those. It will help him winning some time on your site not trying outdated URL. |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche
Last edited by dcz on Sat Jul 08, 2006 8:01 pm; edited 2 times in total |
|
| Back to top |
|
|
Peter77
phpBB SEO Team
Joined: 10 May 2006
Posts: 512
Location: Michigan
|
| Posted: Mon May 29, 2006 6:25 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
I removed all viewforum.php, viewtoic.php from google links and cache... seems like I somehow got rid of info: and entire cache from my site. does it take a really long time for google to start using the SEO Optimized links?
| Code: | | Disallow: /index.php? |
as in disallowing the forum to get indexed? |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
| Posted: Mon May 29, 2006 6:30 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
| Peter77 wrote: | I removed all viewforum.php, viewtoic.php from google links and cache... seems like I somehow got rid of info: and entire cache from my site. does it take a really long time for google to start using the SEO Optimized links?
|
It will be faster than the first time because you already have Backlinks and PageRank. I'd say a month or two for all of your url, but it depends on Google's mood. How often does Google-Bot crawls your site ?
| Peter77 wrote: |
| Code: | | Disallow: /index.php? |
as in disallowing the forum to get indexed? |
No, only index.php?...something... This is to take care of any possible var that could still get posted here, categories, things like that.
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
Peter77
phpBB SEO Team
Joined: 10 May 2006
Posts: 512
Location: Michigan
|
| Posted: Mon May 29, 2006 7:07 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
| dcz wrote: | | Peter77 wrote: | I removed all viewforum.php, viewtoic.php from google links and cache... seems like I somehow got rid of info: and entire cache from my site. does it take a really long time for google to start using the SEO Optimized links?
|
It will be faster than the first time because you already have Backlinks and PageRank. I'd say a month or two for all of your url, but it depends on Google's mood. How often does Google-Bot crawls your site ?
|
Oh that's right... I think you mentioned that before. I do have backlinks and my PR is still at 2. Google visits everyday even if I add "visit after 3 days" ect in meta.
| dcz wrote: |
| Peter77 wrote: |
| Code: | | Disallow: /index.php? |
as in disallowing the forum to get indexed? |
No, only index.php?...something... This is to take care of any possible var that could still get posted here, categories, things like that.
++ |
ah okay, thanks. |
Last edited by Peter77 on Wed May 31, 2006 6:38 pm; edited 1 time in total |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
| Posted: Mon May 29, 2006 8:36 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
amir abbas here are some rewriterule you will enjoy very much 
| Code: |
RewriteRule ^forum-([0-9]+)\.html /forum$1.html [R=301,L]
RewriteRule ^about([0-9]+)-([0-9]+)\.html /topic$1-$2.html [R=301,L]
RewriteRule ^about([0-9]+)\.html /topic$1.html [R=301,L] |
Put those before the other rewriterules.
The principle is simple, if someone hits www.example.com/aboutxx.html it will b redirected with a nice 301 to www.example.com/topicxx.html and the same apply with pagination (aboutxx-yy.html => topicxx-yy.html with an http 301 header). I did not carry on the QSA flag, because I thought it was better to get rid of as well of any other parameter that could be there (SID, sort order etc ...) and to concentrate to Backlinks on really interesting URL.
I could do the same for paginated forum's URL as well, but it seems to me the able2know website is not using the same URL Standard as the one it offers in the able2know mod rewrite.
Could you post here some sample of paginated forum's URL So I can provide you with more rewriterules
But I doubt those where indexed well before. |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
| Posted: Tue May 30, 2006 11:18 am Post subject: Re: how can i remove my old URLs from google database |
|
|
OH
thanks 
it can be very useful |
_________________
چهار گوش - طراحی وب - مجله طراحی وب |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
|
| Back to top |
|
|
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
| Posted: Tue May 30, 2006 12:06 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
thank you
rewrite rules works perfectly
here a sample URL
http://forum.amuzesh.org/about1110.html
there is some other urls
for example
http://forum.amuzesh.org/ntopic1221.html
http://forum.amuzesh.org/ptopic1305.html
i forgot to use this part of able2know mod
| Code: | robots.txt
Disallow: /phpbb/post-*.html$
Disallow: /phpbb/updates-topic.html*$
Disallow: /phpbb/stop-updates-topic.html*$
Disallow: /phpbb/ptopic*.html$
Disallow: /phpbb/ntopic*.html$
|
at this time
i must ban standard URLs with robots.txt file that you gave me
i must fix the WWW prefix with your rewrite rule
finally i must put some rules in robot text for ban ntopic and ptopic pages
and after that i must register this robot.txt file in google |
_________________
چهار گوش - طراحی وب - مجله طراحی وب |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
| Posted: Tue May 30, 2006 12:13 pm Post subject: Re: how can i remove my old URLs from google database |
|
|
If some of those are part of google indexing, and even more if some of them where posted, you can as well keep those backlinks with this :
| Code: |
RewriteRule ^ptopic([0-9]+)\.html /index.php [R=301,L]
RewriteRule ^ntopic([0-9]+)\.html /index.php [R=301,L] |
I redirect those to the forum's index because they are duplicates so we do not want to keep them, but we love to keep the PageRank Transmission to the forum index because it's a good target for the biggest PageRank
| amir abbas wrote: |
| Code: | robots.txt
Disallow: /phpbb/post-*.html$
Disallow: /phpbb/updates-topic.html*$
Disallow: /phpbb/stop-updates-topic.html*$
Disallow: /phpbb/ptopic*.html$
Disallow: /phpbb/ntopic*.html$
|
|
use this instead for those :
| Code: |
Disallow: /phpbb/post
Disallow: /phpbb/updates-topic.html
Disallow: /phpbb/stop-updates-topic.html
Disallow: /phpbb/ptopic
Disallow: /phpbb/ntopic
|
| amir abbas wrote: |
at this time
i must ban standard URLs with robots.txt file that you gave me
i must fix the WWW prefix with your rewrite rule
finally i must put some rules in robot text for ban ntopic and ptopic pages
and after that i must register this robot.txt file in google |
Exactly |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
| Posted: Wed May 31, 2006 7:24 am Post subject: Re: how can i remove my old URLs from google database |
|
|
OK
i set a robots.txt on root of my site
| Code: | User-agent: *
Disallow: /forum/viewtopic.php
Disallow: /forum/viewforum.php
Disallow: /forum/index.php?
Disallow: /forum/posting.php
Disallow: /forum/groupcp.php
Disallow: /forum/profile.php
Disallow: /forum/memberlist.php
Disallow: /forum/search.php
Disallow: /forum/login.php
Disallow: /forum/faq.php
Disallow: /forum/post
Disallow: /forum/updates-topic.html
Disallow: /forum/stop-updates-topic.html
Disallow: /forum/recent.php
Disallow: /forum/dload.php
Disallow: /forum/post |
but some of able2know mod are broken
see this two link
forum.amuzesh.org/about16-0-asc-45.html
forum.amuzesh.org/mark-forum12.html
thank you for youe help
AWstat statistic shows that 37% of page im my site are 404 Error
with this rewrite rules you help me a lot |
_________________
چهار گوش - طراحی وب - مجله طراحی وب |
|
| Back to top |
|
|
JanoF
Joined: 30 May 2006
Posts: 43
|
| Posted: Wed May 31, 2006 9:12 am Post subject: Re: how can i remove my old URLs from google database |
|
|
Is ok this write?
| Code: | User-agent: *
Disallow: /admin
Disallow: /ads
Disallow: /cache
Disallow: /db
Disallow: /images
Disallow: /includes
Disallow: /language
Disallow: /scripts
Disallow: /templates
Disallow: /common.php
Disallow: /config.php
Disallow: /delete_users.php
Disallow: /extension.inc
Disallow: /groupcp.php
Disallow: /index.php?
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /posting.php
Disallow: /privmsg.php
Disallow: /profile.php
Disallow: /search.php
Disallow: /viewforum.php
Disallow: /viewtopic.php
Disallow: /image-* |
|
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14279
|
| Posted: Wed May 31, 2006 10:14 am Post subject: Re: how can i remove my old URLs from google database |
|
|
| JanoF wrote: | Is ok this write?
|
Yes.
For the admin/ folder, you should as well lock it with .htpasswd.
cache/ ? If you are running the phpbbstyles.com Cached Generation (SQL cache) then read this post.
For the db/ and includes/ folder, a good old .htaccess with Deny from all in it is a must as well 
Then I'd just change the "image" line as well as add some other webmedic specific disallows :
| Code: |
Disallow: /image
Disallow: /mark
Disallow: /next
Disallow: /updates-topic
Disallow: /stop-updates-topic
Disallow: /1-
|
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
|
|
