Help me about robot.txt

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Moderator: Moderators

Help me about robot.txt

Postby The_BlacK_MOON » Mon Jan 16, 2012 4:29 pm

I don't have much idea about robot.txt

someone gave me this,

Code: Select all
# User-agent: *
# Crawl-delay: 10

User-agent: *
Host: fjunction.org
Disallow: forum/engine/go.php
Disallow: forum/user/
Disallow: forum/newposts/
Disallow: forum/statistics.html
Disallow: forum/*subaction=userinfo
Disallow: forum/*subaction=newposts
Disallow: forum/*do=lastcomments
Disallow: forum/*do=feedback
Disallow: forum/*do=register
Disallow: forum/*do=lostpassword
Disallow: forum/*do=addnews
Disallow: forum/*do=stats
Disallow: forum/*do=pm
Sitemap: http://fjunction.org/sitemap.xml
Sitemap: http://fjunction.org/rss.xml


Please help me friends, I have no idea :roll:

My phpBB is installed in /forum/ folder, when the site map is generated from /forum/ url.
The_BlacK_MOON
 
Posts: 16
Joined: Sun Jan 01, 2012 8:08 am

Advertisement

Re: Help me about robot.txt

Postby coyote » Tue Jan 17, 2012 5:23 am

That's a robots.txt for different software. Do not use it for phpBB.

Also, do not use the robots.txt you'll find in various places throughout this forum unless you have phpBB SEO installed.
coyote
phpBB SEO Team
phpBB SEO Team
 
Posts: 284
Joined: Fri May 11, 2007 6:24 am

Re: Help me about robot.txt

Postby The_BlacK_MOON » Tue Jan 17, 2012 6:05 am

Wht should i do now? what i need to wright in robots.txt now?
The_BlacK_MOON
 
Posts: 16
Joined: Sun Jan 01, 2012 8:08 am

Re: Help me about robot.txt

Postby coyote » Tue Jan 17, 2012 6:23 am

Do you have phpBB SEO installed? The question is very important.

Look in the contrib folder that came inside the .zip archive. There is one included in there that is appropriate for almost all installations.
coyote
phpBB SEO Team
phpBB SEO Team
 
Posts: 284
Joined: Fri May 11, 2007 6:24 am

Re: Help me about robot.txt

Postby coyote » Tue Jan 17, 2012 6:25 am

It appears that phpBB SEO is not installed on your forum, or not installed correctly.

Do not use a robots.txt file until you get phpBB SEO installed.
coyote
phpBB SEO Team
phpBB SEO Team
 
Posts: 284
Joined: Fri May 11, 2007 6:24 am

Re: Help me about robot.txt

Postby The_BlacK_MOON » Tue Jan 17, 2012 6:31 am

No, I want to install SEO, where is the download link? Im searcing around this forum and getting more than one MOD :roll:

I have very little idea abot SEO, so what I need to install 1st? please help
The_BlacK_MOON
 
Posts: 16
Joined: Sun Jan 01, 2012 8:08 am

Re: Help me about robot.txt

Postby coyote » Tue Jan 17, 2012 6:58 am

Well, you only have 21 posts and 12 members. I suggest installing the phpBB SEO Premod and then do the file edits for the Arty style that you will find in the contrib folder of the .zip archive.

I have to warn you though, if you've never done file edits you're going to be confused about editing that Arty style. You'll want to use the prosilver code changes, but not everything is going to match up 100%.
coyote
phpBB SEO Team
phpBB SEO Team
 
Posts: 284
Joined: Fri May 11, 2007 6:24 am

Re: Help me about robot.txt

Postby The_BlacK_MOON » Tue Jan 17, 2012 7:45 am

Thanks for your help friend :) I will change style if i needed. But 1st of all I must learn somting about phpBB-SEO.

Im doing it right now, Will take supports from you people after having some knowledge. Its tough to get support as a noob. Thanks you :D
The_BlacK_MOON
 
Posts: 16
Joined: Sun Jan 01, 2012 8:08 am

Re: Help me about robot.txt

Postby coyote » Wed Feb 01, 2012 6:52 am

You're very welcome! :D
coyote
phpBB SEO Team
phpBB SEO Team
 
Posts: 284
Joined: Fri May 11, 2007 6:24 am


Return to roBots

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 3 guests