WARNING

1.2.0RC1 is out : To update please read this thread.

What follows is kept for archive.

0.0.2 support will continue for some time for 0.0.2 user, until they update.

Do not use this version if you are first installing this mod.

---------------------------------------------------------------------------------------

mx Google Sitemaps

---

Module name : mx Google Sitemaps

Module author : dcz / www.phpBB-SEO.com

MOD Description : This mod creates Google xml sitemaps for phpBB and mxBB Portal (if used).

Module version : v1.0.1

Installation Level: Easy for the Basics, Advanced if setting up mod_Rewrite for Google Sitemaps.

Installation Time : 5 Minutes for the Basics, a bit more if setting up mod_Rewrite for Google Sitemaps.

Author's Notes :

---

Modular Google Sitemaps Solution for phpBB and mxBB Portal.

This mod will create a SitemapIndex, listing all the created Sitemaps.

Please make sure mxGoogle Sitemaps is working before you submit the sitemap.php file (or sitemaps.xml with mod Rewrite) to the Google Sitemaps Service.

This mod was first inspired by jhaskins's Google sitemap generator (Joe Haskins ~ http://www.streetrod3.com) but is doing a totally different work now.

The code is meant to be as fast and light as possible while giving the appropriates info's (URL and Last Modified Time). The template system is totally bypassed using echo, for a smother and much less ram consuming output allowing for numerous amount of topics to be listed and making the system ready for large boards.

So far, the code did very good job listing up to 5000 unique topics (meaning even more links since the mod outputs topic pagination as well) in a forum sitemap.
Should do it for even more, the Google sitemap standards allows up to 50 000 URLs per sitemap.
And since listing 50 000 topics URL would mean to use quite some resources and to dig out very old threads, you can as well limit the absolute number of outputted links, so that in all cases you will be able to use and take advantage out of this code.

Now why such a system :


You may have noticed Google Bot, as well as other Search Engine Bots, are spidering your pages.
To do so they must first find a link to your page, which will be the home page at first in most case. Then, they'll eventually some find out links to other pages of your site.
Usually, they will come back a couple time before they will start to follow the new discovered links and will continue to 1) dicover new links in pages, and 2) explore new pages.
Now you understand this is taking some time. There are a lot of pages in the internet, and they need quite some visit in the end to find out all of your content, and even more is there are several level to go through before finding links to the actual search-able content.

With phpBB, interesting search-able content is to be found in topics. One have to go to the index to find links to forums, then to forums to find links to topics, at least the good ones (with the best url). And since forums are paginated, there are again other links to go through to spider the site.
So it can be quite long for a Bot to find all of your content.

This is why such a system exists, to provide a lot of links to the Google Bot at once. Everybody is winning, less visit are needed to just find link to search-able content and visits are a lot more efficient.

With this mod, the Google-Bot will visit the SitemapIndex, the only file to submit, and will find out you have one Sitemap per forum and one for forums, with a last modified date based on the last post made in those.
So it will only load the Sitemap with new content and find out up to 50 000 url at a time per sitemap and again, with a last modified date based on the last post of every thread, so it won't even have to re visit all threads before finding the new ones.

By default, new content appears at the top of every list, so one could, let's say, limit the number of outputted links (configurable in ACP) to the last 1000 active thread from each forum only, in order to save some server resources while still providing a reasonable amount of links.
But as well ordering them the other way, older first (configurable in ACP), could be useful for someone with a big board who'd want to list all of his site's content after he decided himself to get rid of SIDs in order to have his old content indexed as well. At least for a while, before coming back to the default sorting and put the priority on new content again as it should.

A new functionality was added to 1.0.1, you can now exclude public forums from the listings. As well, when mxBB or KB is installed, you can now exclude public mxBB pages and/or Kb categories. Can be handy when you think some of those should not be that optimized.

With KB, it's the same, one Sitemap per category listing approved and public articles. Last modified time is based on last post time in forum and comment thread.

Running mxBB Portal, a Sitemap listing the portal's pages will show up in the Sitemap Index, but without last modified date, since it does not apply here (There is no info about this in mx tables for now).

As a conclusion, this mod will not really help to increase Page Rank (PR) since links in those are not considered as Back-links, but it will help a lot to get indexed. The vast majority of the Google-Bot visits will now be used to cache pages instead of just trying to find new content. More page cached in less visit and new content found faster.
And if you just post a link to it somewhere (not to many though, you don't want too many visits on those if you list a lot of links) you will see many other bots exploring it.

This system is 100% stand alone (does not change a single phpbb line) and is totally self installing. Updates are easy as uploading files (and eventually, but I doubt it) run a sql script.

There should be no more issues, if ever was, using ORACLE and POSTGRE.

If you specific Rewrited url standard is currently not supported by the phpBB SEO mod Rewrites, please post a request for it in the phpBB mod rewrite Forum and the Team will provide premodded files.

---

Supported Languages :

English, French.

Demo :

phpBB "Vanilla" Installation
SitemapIndex
Forum Map
The Forum n°2 Map
Preview mx Google SiteMaps ACP

Running mxBB :

Download File :

mx Google Sitemaps V 1.0.1

Updates :

• phpBB :
  To update from RC1 or RC2, you will have do the same as for a new install.
  You will have to re-enter you settings in ACP this time.
• mxBB :
  To update from RC1 or RC2, you will have to uninstall the module in ACP before you do the same as for a new install.
  You will have to re-enter you settings in ACP this time.

Current Features :

Sitemaps Created :

One SitemapIndex listing all sitemaps
One general forum sitemap, listing public forums url
One sitemap per public forum listing topics.


When kb is installed : (both on phpbb and mxBB PORTAL)


• Adds all necessary entries in the SitemapIndex
  
• One general category sitemap, listing all public categories
  
• One sitemap per category, listing approved articles.
When mxBB installed :


• Adds an entry in the sitemap index
• Adds proper otpion for Kb to run with mx in acp (define mx kb page's ID)
• One sitemap listing all public mx pages.
Optimized code :
All major queries are separated into several cycles, configurable in ACP.
Possibility to limit the number of url outputed, configurable in ACP.
Possibility to output (and limit) paginated topics url, configurable in ACP. Pagination will be limited like this :


Code: Select all

"begin"               "end"
    |                    |
    |   Down    Up       |
    |     |     |        |
    v     v     v        v
    1, 2, 3 ... 10, 11, 12


Only public and valid content is listed. 100% Valid output.

Planned features :

Features :
Please suggest, cache feature maybe, support for Yahoo standard.

Other Mods support :
Will see if other mods would benefit such a listing.

Other mod_Rewrite support :
Will add premoded files in the package for any unsupported URL standard upon request.

Contrib :

Pre modded files for several mod rewrite included.
If apply, just ovewrite the standard files with the right contrib files.
Supported url standars : phpBB SEO mod Rewrites, Webmedic's rewrite mod, able2know rewrite mod, GoogleBB Links ...

Support :

English Support On this Thread. If you need some specific adaptation, please start a new thread in this Forum.
Support Francophone

Preview :

Here is what you will see ACP side. Note that in this example, mx Sitemaps is not Installed, the mx Google Sitemaps module will only use and install the Google SiteMaps menu. The mx SiteMaps menu will show up in the same menu category if installed.

Note : An extra option shows up while runnig mxBB Portal, allowing to set up the KB mxBB page ID and to set up proper exclusions.

ACP Menu :

---

Mx SiteMaps

Google SiteMaps

SiteMaps :

---

Google Sitemaps

The Google sitemap system allows GoogleBot to find pages far away from the Home page easyer. This system generates a sitemapp index pointing to the different sitemaps available.
You must register your sitemapIndex @ Google if you want to access some interesting stats.
You can proceed anonymous though

Google Sitemaps Settings
SQL cycle Major queries are sparated into several cycles in order not to overload the SQL server. This is the maximum number of topics to fetch within a single query
Url Limit Maximum number of url outputed in each sitemap. This limit being checked in every SQL cycle, the actual outputed number of url is this limit +- 1 SQL cycle +- number of paginated topics (limited or not) in the last cycle. Limited by default to 40 000, knowing Google will go up to 50 000 per sitemap file.
Sort Order All outputed links are sorted in the same way topics are sorted by default in phpbb (last activity DESC). You can set this to DESC for example if you whish to make it easyer for Google to find again links to archeological or locked threads (eg inactive for a looong time).	DESC   ASC
Mod Rewrite Sitemaps If activated, the sitemap's url will be rewrited. CAUTION : you MUST run Apache server with mod rewrite activated and set up the .htaccess located in this release's contrib/ folder properly. NOTE : This will only affect the sitemaps url provided in the sitemap index. There is no problem for google to visit non url rewrited sitemaps.	No   Yes
Forums Sitemaps Settings
Forum Exclusions You can exclude some public forums from the mx Google Sitemaps Listing. Enter the exclude forum IDs list, coma separated : e.g 1,5,8. Note : If not filled, all public forums will be listed.
Announcement Priority Announcement Priority (must be a number between 0.0 & 1.0 inclusive)
Sticky Priority Sticky Priority (must be a number between 0.0 & 1.0 inclusive)
Default Priority Priority for regular topics (must be a number between 0.0 & 1.0 inclusive)
Topic Pagination: Low Limit Paginated topic link output is handeled. Enter here how many paginated topic pages, from the begining, are to be outputed. If set to 0, it won't output paginated links after the first topic page.
Topic Pagination: Upper Limit Enter here how many paginated topic pages, starting from the last one, are to be outputed. If set to 0, it won't output paginated links before the last topic page.

Adding Content in mx Google Sitemaps listings :

Here is a small and handy add on that will help out a lot.

The principle is very easy, you just have to upload three files in mx_ggsitemaps/includes :

index_site.php
sitemap_site.php
url_list.txt

An new URL will be listed in your Google SitemapIndex :

http://www.example.com/sitemap.php?site

This new Google Sitemap will just output the URLs you'll put in url_list.txt, one per line, and formatted as follow :

http://www.example.com/url.html

To make it short, you'll have to upload the three files and to put the URL list you want to output in url_list.txt et voilà

Note :

URLs listed in url_list.txt must be part of the same domain (or sub-domain) as the SitemapsIndex (sitemap.php) and shall not be located above in the folder structure.

If sitemap.php is located in : www.example.com/dossier/sitemap.php

You can add :

http://www.example.com/dossier/*.*

But you can't add :

http://www.example.com/*.*

Nor :

http://sous-domaine.example.com/*.*

you can theoretically list up to 50 000 URLs in url_list.txt, but, without test performed above few hundreds, I think you should not go over to 2 or 3000 URLs.

You can easily clone this add on following these easy steps :

1. Rename the three file to :
   
   
   index_site1.php
   sitemap_site1.php
   url_list1.txt
2. Open : index_site1.php
   
   Replace :
   

   Code: Select all

      echo "\t\t" . '<loc>' . $root_url . 'sitemap.' . $phpEx . '?site</loc>' . "\n";

   
   With :
   

   Code: Select all

      echo "\t\t" . '<loc>' . $root_url . 'sitemap.' . $phpEx . '?site1</loc>' . "\n";

3. Open : sitemap_site1.php
   
   Replace :
   

   Code: Select all

   $site_map = ( isset( $HTTP_GET_VARS['site'] ) ) ? TRUE : FALSE;

   
   With :
   

   Code: Select all

   $site_map = ( isset( $HTTP_GET_VARS['site1'] ) ) ? TRUE : FALSE;

   
   Replace :
   

   Code: Select all

      $url_array = @file($module_root_path . "mx_ggsitemaps/includes/url_list.txt");

   
   With :
   

   Code: Select all

      $url_array = @file($module_root_path . "mx_ggsitemaps/includes/url_list1.txt");

This will allow you to add as many custom lists as you want.

They'll be accessible from http://www.example.com/sitemap.php?site1

This way, it's useless to build up too large url_list.txt, it's more efficient to build several if you need it (url_list1.txt url_list2.txt ... url_listN.txt).

To make it easier : SiteSpider.

This free software, as wel as many other arround, will, with some setup, output a noce URLs list, txt format, onbe per line

Don't forget it's useless to list URLs already listed in the regular sitemaps
Many criteria are available to filter spidered URLs using SiteSpider.

If you need help using or setting up SiteSpider, please post a new thread (or use the existing ones) in the SEO Techniques Forum

Other technique are available to build such URL listings, do not hesitate to share the one you like here

Download the Pack : mx Google Sitemap txt plug-in

++

It's probably mentioned somewhere, but I can't find it:

What's the main difference between mx Google Sitemaps and mx SiteMaps Module?

I'm running phpBB-SEO-Advanced on my forum. Which SEO sitemap do you recommend?

Well,

mx Google Sitemaps creates specific sitemaps for Google. As you can see in the demo listings, this is an xml output, quite simple, but very efficient.
This is meant to allow Google to find up to 50 000 URLs in one shot

mx Sitemaps is a regular site map module, Search Engine Optimized, listing links in html format.
It's meant for human users, they'll be able to track new messages and such in it and will see what they have the right to.
The SEO part of it is as for any regular site map, providing links to your content, here the topic and forums.
The cookie being, beside this mod will again output no dupes, the highly dynamic output, thanks to the last active topic listing, upon almost every visit, bot will see content has changed on all pages

Take your time to look at the demo in the other thread, you can as well add new links in the index from acp, very handy.

++

dcz wrote:[...]

Ah, I don't need any user-friendly sitemaps, so I'll stick with mx Google Sitemaps mod which was already running on my website before I installed the phpBB-SEO-Advanced mod.

Another question:
The sitemap-index is located in: /sitemap.php

This sitemap-index contains links to the sitemaps, with urls like:
<sitemap>
<loc>http://XXXXX/sitemap.php?fid=81</loc>
<lastmod>2006-06-13T11:47:59+00:00</lastmod>
</sitemap>

And http://XXXXX/sitemap.php?fid=81 contains urls like:
<url>
<loc>http://XXXXX/viewforum.php?f=81</loc>
<lastmod>2006-06-13T11:47:59+00:00</lastmod>
<changefreq>always</changefreq>
<priority>1.0</priority>
</url>

But it would be better if it's pointing to the phpBB-SEO-Advanced urls, right?
So http://XXXXX/viewforum.php?f=81 should be something like:
http://XXXXX/forum-title-vf81, right?

How can I accomplish this?

To do this, you just need to use the premodded files located in the contrib/ folder of the release instead of the default ones.

So together with the phpBB SEO Advanced mod Rewrite, you should use :

contrib/moded_4_mod_rewrites/phpBB_SEO_mod_Rewrites/Advanced_Mod_Rewrite/sitemap_forum.php

instead of the one located in your ftp (mx_ggsitemaps/includes/).

Just overwrite it and it will be ok.

Then, you can either submit sitemap.php to Google, or, if you prefer, can set up mod rewrite for the Google sitemaps as well (in ACP), but this is mostly a cosmetic feature, Google will spider sitemaps the same.
Those pages won't have PageRank anyway so ...
To do this you'd have to install as well the additional rewriterules for the new sitemaps links in the contrib/.htaccess file of the release.

Once you are able to follow links and to find the proper URLs for topic and forums, you can submit

Then, mx sitemaps is not only user-friendly, it's helping out your topic's PageRank (creates more good links to them) and spidering, since bot will have many more occasions to find contents.

Actually, a site map is a must for any type of sites, and mx Sitemaps is an Search Engine Optimized site map solution for phpBB forums.

As always, everyone decides

++

Updated to 1.0.1

Added some more options, such as the ability to exclude public content from the listings.

Should be no more problem, if ever was with ORACLE and POSTGRE.

++

how we can update ?

there isn't any db_update.php file in new pack

Hey amir
read the mx_ggsitemap_V1.0.1.txt

=== > mxBB PORTAL Update <===
_____________________________

To update from RC1 or RC2, you will have to uninstall the module before you do the same as for a new install.
You will have to re-enter you settings in ACP this time.

....

===> phpBB Update <===
______________________

To update from RC1 or RC2, you will have do the same as for a new install.
You will have to re-enter you settings in ACP this time.

greetings
macnack

thanks macnack

but by running db_update.php file you will lose your configuration
the number of items in ACP for google sitemap is not very high but this method for updating is not very good

anyway thanks

amir abbas wrote:thanks macnack

but by running db_update.php file you will lose your configuration
the number of items in ACP for google sitemap is not very high but this method for updating is not very good

anyway thanks

Jeez, I made you loose 2 minutes

I just though an upgrade script was not worth it this time, but I will build one for mx Sitemaps where there are a lot more things stored in db.

The update is decently easy though, will do better next time

++

DCZ wrote:Jeez, I made you loose 2 minutes

i offered only a simple suggestion
i thought that you forgot to put update db file in new pack

thanks

Hello, first off thanks for this great mod. I have just have a small issue. Now the sitemap.php works as you can see here

My Sitemap

Now I have the url rewrite on from the ACP which it does rewrite, but the links it rewrites are not working

I.E

http://www.texascampingforum.com/forum/forum-sitemap-33.xml


So is their a something else I need to do or a certain sitemap_forum file I need to upload? I am using a SEO rewrite-mod to convert php to static html but not any from here because I am using PHPBB Plus which it already had installed.

And welcome

Actually, it seems you are using the phpBB SEO simple mod rewrite.

So, you should upload the premodded file for it, located in the contrib/ folder and use it instead of the default one (so that in such links you see the correct URLs : -http://www.texascampingforum.com/forum/viewforum.php?f=12

Now, for the mod rewrite for the mx Google sitemaps links, you just need to implement the rewriterules located in the contrib/ folder's .htaccess in your form's .htaccess (just the rewriterules, no need to start the rewrite engine twice )

Once done, the rewritten links (activated in mx Google stemaps ACP will work.

Once you see the links outputted are following your URL standard (the simple one it seems) then you can submit it .

And don't forget the final touch

Then as an SEO Advise, I think you could open you index to guest, is it a performance issue ?

++

dcz wrote:Then as an SEO Advise, I think you could open you index to guest, is it a performance issue ?
++

The forum permissions are actually open. I just installed a mod that requires uses to login or force them to create an account. I found once I did that the members went up. Kind of like a curosity kills the cat theory. Not to mention I am monitioring who is vewing the forum as their is another group I am trying to keep out and from viewing the content.

simple mod rewrite works but only if I turn the rewrite off in the ACP. It will produce the site map in a dynamic URL. but it referrs to links that are static so I guess that should work for now and we will keep an eye on things.