configure web crawler for my web site?

Discussions about the phpBB2 Forum. How to get the best from this powerful script.

Moderator: Moderators

configure web crawler for my web site?

Postby sovanndy » Fri Jun 20, 2008 12:38 pm

Could anyone explain me the step of how to configure or make web crawler for my forum phpbb3 web site?
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Advertisement

Postby TomaS » Fri Jun 20, 2008 1:52 pm

just use permmison in bb3 system
phpBB podpora-slovak phpBB support
slovenský preklad pre phpbb3-slovak translate for phpBB3
predaj a kupa domeny, marketing a internetove podnikanie- Slovak domain center
TomaS
PR2
PR2
 
Posts: 229
Joined: Fri Jun 08, 2007 1:22 am

just configure permission in phpbb3 system is enough?

Postby sovanndy » Fri Jun 20, 2008 3:00 pm

Thank Tomas

Is just using permission in PHPBB3 system enough? if so, what I have to configure in phpbb3 permission system?

Don't I need to register account for Googlebot of Google Search engine and add my website url to that search engine, or to create sitemap, Robots for Google, MSN, Yahoo or other Search Engine?


Best Regards,
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Postby SeO » Sat Jun 21, 2008 8:56 am

So you don't want to actually make a crawler but rather to allow the search engines to crawl your forum ?

First, about the phpBB bots handling, my advice is to barely not use it, it's ok for SID removing, but it's technically cloaking, so I would not go too far into configuring custom authorisations for bots.

Personally I set bots authorisation the same as guests, it's safer and usually there is no real need to go further.

About what you could do SEO phpBB, oh well, we've released several useful mods for this purpose here.

You could start playing with our SEO premod to see all our mods at once without code editing.

From there you could either use it or use your selection of phpBB SEO mods. Mod rewrite and zero dupe are really useful, title optimization and dynamic meta tags are too ;)
SeO
Admin
Admin
 
Posts: 6334
Joined: Wed Mar 15, 2006 9:41 pm

making Spiders, Bots, or Crawlers?

Postby sovanndy » Sat Jun 21, 2008 9:34 am

I would rather to use crawlers and your ways....

I could would rather to use both ways to make Crawler, Bots, or Spiders?

Could you explain me how to do that?

Best regards,
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Postby SeO » Sat Jun 21, 2008 9:41 am

I'm not sure to get it here.

You would like to build your own crawler ? For what purpose ?
Or am I missing something ?
SeO
Admin
Admin
 
Posts: 6334
Joined: Wed Mar 15, 2006 9:41 pm

Crawling page?

Postby sovanndy » Sat Jun 21, 2008 4:08 pm

No I mean that I would like to create Sitemap.xml file, Robots.txt file in my web site directory by combining with your SEO premod to completely reach crawling-page Search Engine configuration for my web site.

My aim is that if someone post a new topic in my forum web site, then it will promptly show in Search Engine when someone surf something related to the content of old and recently-posted topic. that is the target I have to do .

Because I heard Google said that if someone would like Search Engine to crawl their web site content, they have to follow some steps, first go to register googlebot account by adding their web site url, second create sitemap.xml file after that....

That is what I would like to do, called "making crawler for my web site." in subject in this forum.


so Could you explain me all the steps to completely reach that? and What else do I have to do in PHPBB3.0.1 configuration?

Thank in advance
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Postby SeO » Sat Jun 21, 2008 4:50 pm

For the robots.txt, the SEO premod and our mod rewrites comes with one to implement (read the warnings in install for special cases).

For Google sitemaps time will come

http://phpbb3.phpbb-seo.net/sitemapindex.xml

... soon ;)
SeO
Admin
Admin
 
Posts: 6334
Joined: Wed Mar 15, 2006 9:41 pm

bots permission?

Postby sovanndy » Tue Jun 24, 2008 6:21 am

what is bots permission in PHPBB3.0.1 configuration?

why we should configure bots permission?


You kind will be highly appreciated.
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Postby SeO » Tue Jun 24, 2008 7:35 am

It's for bots, personally I set them to match the guest auth, I advise you not to go too far playing with phpBB3 bots auth, one shall not forget it's cloaking ;)
SeO
Admin
Admin
 
Posts: 6334
Joined: Wed Mar 15, 2006 9:41 pm

cloaking???

Postby sovanndy » Tue Jun 24, 2008 3:06 pm

if I play with Bots or Spider, Will your re-write mod affect Bots or Spider or not,which they think it's like cloaking?
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Postby SeO » Wed Jun 25, 2008 8:04 am

Our mods do not cloak, phpBB3 does.
SeO
Admin
Admin
 
Posts: 6334
Joined: Wed Mar 15, 2006 9:41 pm

problem with phpbb3 bots authentication?

Postby sovanndy » Wed Jun 25, 2008 11:52 am

Why you said that too far playing with PHPBB3 bots authen will go into cloaking?

What is the problem with PHPBB3 authentication?
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Postby SeO » Thu Jun 26, 2008 8:02 am

Well because phpBB3 is cloaking when doing things for bots only. It's not about auth, it's about bot's auth.
SeO
Admin
Admin
 
Posts: 6334
Joined: Wed Mar 15, 2006 9:41 pm

GYM Mod support with rewrite mode?

Postby sovanndy » Sat Jun 28, 2008 2:40 pm

Does GYM Mod will detect or support rewrite mod?

and What is different between Mx_sitemaps and GYM mod?

are they stable to use?

if someone post new topic to forum, how long will it add new link to sitemap file, and how long will sitemap file be submitted to google?
sovanndy
PR0
PR0
 
Posts: 85
Joined: Tue Jun 10, 2008 8:11 am

Next

Return to phpBB2 Forum

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 6 guests