Hi, I have my root folder as public_html where all my phpbb files are lying, and I would like to know how do I make a good robot.txt and what to disallow etc.

I have this on my robot.txt

Code: Select all

User-agent: *
Allow: /

see here:
http://www.gnc-web-creations.com/creati ... t_file.htm

also u can use the generators like:
http://www.mcanerin.com/en/search-engine/robots-txt.asp

Thanks, and what files to recommend in phpbb that should I put disallow?

Taken from phpBB SEO Premod V 3.0.6

• robots.txt
  A sample robots.txt is provided in the package to make sure no useless content is taken into account by the Search Engines.
  Warning :
  
  • In case you are migrating from a previous unrewritten phpBB version, wait couple month after you activate the zero duplicate mod before you implement the robots.txt. In such case we want to redirect first and then drop useless content.

Direct link to sample robots.txt
http://phpbb-seo.coda-cola.net/svn/phpb ... robots.txt