Hi!

I was just experimenting with robots.txt and wanted to know whether I should disallow /member and /messages in phpbb3 seo premod. I want search engines to view users profiles and their messages. Is there anything wrong with seo if I do this? I don't want to do anything bad with search engines and seo. What will be the best from seo point of view?

It's mostly as you wish, but profile pages do not usually have enough good content to be worth showing up on serps.

This of course depends on how important profiles are on your board, but the first target is the topic for SEO.

Code: Select all

User-agent: *
Disallow: /adm/
Disallow: /cgi-bin/
Disallow: /cache/
Disallow: /docs/
Disallow: /download/
Disallow: /files/
Disallow: /images/
Disallow: /includes/
Disallow: /language/
Disallow: /phpbb_seo/
Disallow: /store/
Disallow: /styles/
Disallow: /common.php
Disallow: /common.php?
Disallow: /config.php
Disallow: /config.php?
Disallow: /cron.php
Disallow: /cron.php?
Disallow: /download.php
Disallow: /download.php?
Disallow: /faq.php
Disallow: /faq.php?
Disallow: /fs.php
Disallow: /fs.php?
Disallow: /index.php
Disallow: /index.php?
Disallow: /index.html?
Disallow: /mcp.php
Disallow: /mcp.php?
Disallow: /memberlist.php
Disallow: /memberlist.php?
Disallow: /posting.php
Disallow: /posting.php?
Disallow: /ratetopic.php
Disallow: /ratetopic.php?
Disallow: /report.php
Disallow: /report.php?
Disallow: /search.php
Disallow: /search.php?
Disallow: /style.php
Disallow: /style.php?
Disallow: /ucp.php
Disallow: /ucp.php?
Disallow: /viewforum.php
Disallow: /viewforum.php?
Disallow: /viewonline.php
Disallow: /viewonline.php?
Disallow: /viewtopic.php
Disallow: /viewtopic.php?
Disallow: /messages
Disallow: /post
Disallow: /?

Well! It's kind of wired but I really have disallowed each and everything from my forum. I've allowed bots and guests to crawl /members/seo (example) area. I've redirected index.php to index.html in .htaccess. Please let me know if I'm doing something wrong. I only want search engine to find .html files (topics and members) nothing else. Do I need to do any changes in above robots.txt file?

To do what you want, the suggested robots.txt is enough.

The robots.txt is just a suggestion, you are not actually protecting your files doing so, and actually, disallowing file that are never accessed is a way to have bots trying them (otherwise they would not know about them).

Only Disallow: /images/ could have some interest if you want to make sure bots won't list images directly from this dir.