| |
| |
|
|
|
|
| |
|
| |
|
| :: |
| Author |
Message |
apeee
PR0
Joined: 06 Nov 2007
Posts: 60
|
 Posted: Tue Dec 11, 2007 9:06 pm Post subject: what to block and what not to in robots.txt |
|
|
1. I'm using | Quote: | | RewriteRule ^index\.html$ /index.php [QSA,L,NC] |
in .htaccess, should I disallow index.php like Disallow: /forums/index.php? or Disallow: /forums/index.php
2. Why are you disallowing faq.php in robots.txt?
3. Can i also disallow all the directories like adm, cache, cgi-bin, docs, download, files, images, includes, install (deleted), includes, language, store, phpbb_seo? (all directories)
4. Anyway, this can be the silliest question ever, where are all these .html generated files are stored? some specific directories?
5. Can I rewrite unanswered posts and others to .html?
____________________________
One important thing: when I validate my website for links through w3.org, the link of your website kept in footer "http://www.phpbb-seo.com/" gets error like
| Code: | http://www.phpbb-seo.com/index.php
What to do: The link is forbidden! This needs fixing. Usual suspects: a missing index.html or Overview.html, or a missing ACL.
Response status code: 403
Response message: Forbidden
Line: 216 |
Are you blocking incoming links? why? |
|
|
| Back to top |
|
 |
|
|
Peter77
phpBB SEO Team
Joined: 10 May 2006
Posts: 512
Location: Michigan
|
| Posted: Tue Dec 11, 2007 9:35 pm Post subject: Re: what to block and what not to in robots.txt |
|
|
You can use
Dissallow: forums/index.php? because since forums/index.php is 301 re directed to forums/ anyway.
2. Generally, you want to disallow pages that do not add much value to your site as far as content wise. faq page is important page, but to your members only.
Please note that Google warns of having too many lines in your robots.txt. I believe the limit is 100 lines.
3. You can disallow more directories and pages that you wish, yes.
I have extra directories in my robots text such as modules, images, cache, etc.
The html pages are not stored on your site, actually. they are generated thanks to the rewrite rules in your .htaccess
I'm moving this to the "robots" forum. It sounds like you are running phpbb3, but since your question and topic are geard more towards robots.txt ... |
|
|
| Back to top |
|
|
|
| Navigation |
Similar Topics |
|
|
|
|
|
|
|
| |
|
|
|
|
| |
|
|
|
|
| |
