Best Robot.txt for phpBB3.01

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Moderator: Moderators

Best Robot.txt for phpBB3.01

Postby darkone » Wed May 14, 2008 11:37 am

Still somewhat new to working on the robots.txt file and sometimes I don't even think it works? I add profile.php to the list and then I observe a bot looking at the profiles :)

I know the system isn't perfect but is there a standard format for phpbb3.01 that provides the best security for users on the forums but also make the site indexable?

Can someone post an example of a robot file that maximizes user/site security but at the same time maximizes index ability of your forums?


Thanks

-D1-
darkone
 
Posts: 21
Joined: Thu Jul 12, 2007 3:23 pm

Advertisement

Postby Professional » Wed May 14, 2008 4:07 pm

yes can anyone plz explain about it? :oops:
User avatar
Professional
PR5
PR5
 
Posts: 543
Joined: Mon Apr 07, 2008 5:41 am
Location: 1/2 of the World


Return to roBots

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 1 guest