configure web crawler for my web site?

[sovanndy]

Could anyone explain me the step of how to configure or make web crawler for my forum phpbb3 web site?

[TomaS]

just use permmison in bb3 system

[sovanndy]

Thank Tomas

Is just using permission in PHPBB3 system enough? if so, what I have to configure in phpbb3 permission system?

Don't I need to register account for Googlebot of Google Search engine and add my website url to that search engine, or to create sitemap, Robots for Google, MSN, Yahoo or other Search Engine?


Best Regards,

[SeO]

So you don't want to actually make a crawler but rather to allow the search engines to crawl your forum ?

First, about the phpBB bots handling, my advice is to barely not use it, it's ok for SID removing, but it's technically cloaking, so I would not go too far into configuring custom authorisations for bots.

Personally I set bots authorisation the same as guests, it's safer and usually there is no real need to go further.

About what you could do SEO phpBB, oh well, we've released several useful mods for this purpose here.

You could start playing with our SEO premod to see all our mods at once without code editing.

From there you could either use it or use your selection of phpBB SEO mods. Mod rewrite and zero dupe are really useful, title optimization and dynamic meta tags are too

[sovanndy]

I would rather to use crawlers and your ways....

I could would rather to use both ways to make Crawler, Bots, or Spiders?

Could you explain me how to do that?

Best regards,

[SeO]

I'm not sure to get it here.

You would like to build your own crawler ? For what purpose ?
Or am I missing something ?

[sovanndy]

No I mean that I would like to create Sitemap.xml file, Robots.txt file in my web site directory by combining with your SEO premod to completely reach crawling-page Search Engine configuration for my web site.

My aim is that if someone post a new topic in my forum web site, then it will promptly show in Search Engine when someone surf something related to the content of old and recently-posted topic. that is the target I have to do .

Because I heard Google said that if someone would like Search Engine to crawl their web site content, they have to follow some steps, first go to register googlebot account by adding their web site url, second create sitemap.xml file after that....

That is what I would like to do, called "making crawler for my web site." in subject in this forum.


so Could you explain me all the steps to completely reach that? and What else do I have to do in PHPBB3.0.1 configuration?

Thank in advance

[SeO]

For the robots.txt, the SEO premod and our mod rewrites comes with one to implement (read the warnings in install for special cases).

For Google sitemaps time will come

http://phpbb3.phpbb-seo.net/sitemapindex.xml

... soon

[sovanndy]

what is bots permission in PHPBB3.0.1 configuration?

why we should configure bots permission?


You kind will be highly appreciated.

[SeO]

It's for bots, personally I set them to match the guest auth, I advise you not to go too far playing with phpBB3 bots auth, one shall not forget it's cloaking

[sovanndy]

if I play with Bots or Spider, Will your re-write mod affect Bots or Spider or not,which they think it's like cloaking?

[SeO]

Our mods do not cloak, phpBB3 does.

[sovanndy]

Why you said that too far playing with PHPBB3 bots authen will go into cloaking?

What is the problem with PHPBB3 authentication?

[SeO]

Well because phpBB3 is cloaking when doing things for bots only. It's not about auth, it's about bot's auth.

[sovanndy]

Does GYM Mod will detect or support rewrite mod?

and What is different between Mx_sitemaps and GYM mod?

are they stable to use?

if someone post new topic to forum, how long will it add new link to sitemap file, and how long will sitemap file be submitted to google?