| :: |
| Author |
Message |
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
 Posted: Wed Dec 13, 2006 11:01 pm Post subject: Google caching of phpbb forum with advanced mod rewrite |
|
|
Hi, dcz, I originally posted this in the Google Sitemaps thread and you asked me to start a new thread in here.
| Quote: | How long does it typically take for Google to index a site once a proper sitemap has been uploaded?
In the past it has taken a couple of months from moving to a new domain for the site to be fully indexed and cached.
At the moment the new URLs aren't showing up and the old ones are of course blocked by robots.txt
I have seen GoogleBot spidering my site since uploading the sitemap (and it is regularly checking the sitemap as it should) but it only stays on the main index page at the moment, it doesn't appear to be accessing the other forums yet.
I'm just getting a bit nervous, as all my old URLs are dropping off Google's index. |
You replied:
| Quote: | Did you implemented some HTTP 301 redirection while changing domain ?
If not, it could still be interesting to add some in order to keep the old backlinks.
Anyway, with PR3, it usually takes about one month to see all pages listed in the sitemaps in Google's cache.
The pattern is nothing for a month, and all the sudden, all pages showing up.
If your old URLs are still listed in Google's cache, then, there is a urge to HTTP 301 redirect them.
I was able to change both phpBB SEO forum's URL without even a single visit lost from the Search Engines. So it may be still time to save some.
If it's only phpBB related, please start a new thread in the phpBB forum, if it's about another script, please post in the SEO techniques forum instead.
And we'll see what's doable |
OK, it's now two weeks since the friendly URLs and the sitemap were implemented. So far, the only new URLs showing are forums and threads that happen to be linked to from the home page. I know you said it takes a month, I hope you're right that they will all suddenly appear... 
You asked if I had a 301 redirect from the old domain. I'm not sure if it was a 301 but yes, the old domain had some form of redirect and the old URLs are still indexed.
Google also still has the old URLs from the current domain (viewtopic.php?t=xxx type) indexed as well, but they are no longer cached because GoogleBot is blocked from seeing them by robots.txt.
The problem is that there are a few pages that are still linked to from other sites, so obviously I would like Google to be able to "see" these so I can restore the PageRank for these pages. Is there a way of telling Google that these pages have changed?
So if it follows a link from another site to -http://www.example.co.uk/forum/viewtopic.php?t=285 - is there a way to tell it that the page is now at this link? Some of these older pages have far more external links to them because I have only recently switched to friendly URLs.
Thanks. |
Last edited by arch stanton on Sun Jan 21, 2007 1:13 am; edited 1 time in total |
|
| Back to top |
|
 |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14779
|
| Posted: Thu Dec 14, 2006 12:32 am Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
| arch stanton wrote: | | I'm not sure if it was a 301 but yes, the old domain had some form of redirect and the old URLs are still indexed. |
This you should make sure about. Use http://web-sniffer.net/ to check the header returned. Can you change the dns for the old domain, is it linked to some hosting ?
then, only use this as a robots.txt for phpbb :
| Code: | User-agent: *
Disallow: /forum/viewtopic.php?p=
Disallow: /forum/viewforum.php?mark=
Disallow: /forum/index.php?
Disallow: /forum/posting.php
Disallow: /forum/groupcp.php
Disallow: /forum/profile.php
Disallow: /forum/memberlist.php
Disallow: /forum/search.php
Disallow: /forum/login.php
Disallow: /forum/faq.php |
And I PM you the zero duplicate right now 
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
| Posted: Thu Dec 14, 2006 5:35 pm Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
dcz, as I still own the old domain, I am just using web forwarding from the registry until it expires. I typed the old domain (gladetalk.org.uk) into web-sniffer and got the following response:
| Quote: | <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE>302 Found</TITLE>
</HEAD><BODY>
<H1>Found</H1>
The document has moved <A HREF="http://www.gladtalk.co.uk/">here</A>.<P>
</BODY></HTML>
|
Thank you for the script. What exactly does it do?
And what do I do about old-style gladtalk.co.uk URLs (i.e. ones on the current domain) like the one I mentioned in the first post? These have several inbound links - but of course Googlebot can't see them in this form because viewtopic URLs are blocked in robots.txt. Is there a way of fooling Googlebot into "seeing" them in friendly URL format? |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14779
|
| Posted: Thu Dec 14, 2006 6:13 pm Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
Well, if your old domain had backlinks, it could be interesting to keep it a bit more time and to properly redirect it.
An easy way to do this, since it seems you're paying for it, you should be able to change DNS for it.
The idea would be to link it to your new hosting, if possible.
Then, the www prefix redirection (version 2) would do the rest where your new domain is installed.
The www redirection will as well make sure it's the correct and unique domain to link here, so with it your old one will be redirected.
Talking a bout how long should you maintain the old domain, well it depend, mostly on the backlinks. If not too expensive, I'd keep it at least a year, and even more.
Then, about your first issue, yes the zero dupe will take care of it.
Check the examples in the zero dupe thread
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
| Posted: Thu Dec 14, 2006 6:33 pm Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
dcz, just to clarify re the backlinks to threads on the existing domain:
For example, -http://www.gladtalk.co.uk/forum/viewtopic.php?t=xxx
which Google can't see at the moment.
If I change this line in robots.txt from "Disallow: /forum/viewtopic.php" to "Disallow: /forum/viewtopic.php?p=", will this allow GoogleBot to see the backlinked thread again? And if I run the zero dupes script, this will prevent any dupe penalties? |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14779
|
|
| Back to top |
|
|
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
| Posted: Thu Dec 14, 2006 6:49 pm Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
| dcz wrote: | Exactly, this kind of url will be HTTP 301 redirected, and bots know this means they should forget about the old one and rather use the new one from then.
++ |
Yeah, but if it forgets the backlinked ones, I lose the page ranking for those pages surely?
The thread I mentioned in the opening post has lots of backlinks in the viewtopic.php?t=xxx form but none in the friendly URL form, so it's getting no ranking at all. |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14779
|
|
| Back to top |
|
|
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
| Posted: Sun Dec 17, 2006 11:54 am Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
OK, dcz, have finally implemented zero dupe and changed the robots.txt. Will let you know how I get on.  |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14779
|
|
| Back to top |
|
|
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
| Posted: Mon Dec 18, 2006 10:56 pm Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
| dcz wrote: | To accurately track your results, you can post a small report
++ |
Will do, but in the meantime I'm a bit concerned that GoogleBot is still only looking at the forum index after two weeks. Shouldn't it be looking at the other forums and subforums by now?
Inktomi and MSN have got used to the friendly URLs and are spidering all the forums, so why is Google ignoring the sitemap? |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14779
|
|
| Back to top |
|
|
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
| Posted: Tue Dec 19, 2006 1:25 pm Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
| dcz wrote: | then, only use this as a robots.txt for phpbb :
| Code: | User-agent: *
Disallow: /forum/viewtopic.php?p=
Disallow: /forum/viewforum.php?mark=
Disallow: /forum/index.php?
Disallow: /forum/posting.php
Disallow: /forum/groupcp.php
Disallow: /forum/profile.php
Disallow: /forum/memberlist.php
Disallow: /forum/search.php
Disallow: /forum/login.php
Disallow: /forum/faq.php |
|
dcz, shouldn't I also have forum/member and forum/post disallowed in the robots.txt? |
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14779
|
|
| Back to top |
|
|
arch stanton
PR1
Joined: 04 Oct 2006
Posts: 117
|
| Posted: Tue Dec 19, 2006 10:58 pm Post subject: Re: Google caching of phpbb forum with advanced mod rewrite |
|
|
Oh.
When you said "only use this as a robots.txt for phpbb", I took that literally to mean that only those lines should be in the robots.txt and nothing else.
I have re-inserted the forum/member and forum/post lines. Are there any other ones that I should put back in? |
|
|
| Back to top |
|
|
|
|
