Hello. I installed the mod_rewrite mod. Everything works great, the urls are mod rewrited, but when google visits my site, gives me an error in google webmaster tools. URLs restricted by robots.txt (8) - -http://www.domain.com/post1.html, -http://www.domain.com/post2.html, -http://www.domain.com/post3.html .....

Actually my post urls are not in these format, they are in the format of topic name. Is it problem when google don't index urls like post1.html (blocked by robots.txt)?

This comes from the fact you most likely did not implement the advised robots.txt at first.

It's no big deal, here is how to fix this :
1) drop the post line in your robots.txt in order to allow post urls again (I know they are dupes, but we'll take care of them in step 2) )
2) Install activate the zero duplicate and set post redirection to at least guest (none by default).

This way is faster because Google will continue to crawl post url and will get http 301 redirected to topic urls, where if you keep the disallow, it will not acknowledge the redirecting since it will not (or only very occasionally) crawl these anymore.

After couple month, post url will have disappeared from Google index, then put back the disallow on post urls, this will save some useless redirecting for bots.

++

Hi thanks for the reply. Is it problem to don't touch anything in the code. Because I am not interested in indexing these postXXX.html pages. I want Google to index posts like -http://www.domain.com/some-test-topic-t1.html. If Google give me an error for postXXX.html, then does Google will indexing my mod_rewrited posts (http://www.domain.com/some-test-topic-t1.html) without any problems? Thanks

PS: Offtopic: Does anyone know how to prevent users to delete their own posts? Thanks again

rado84 wrote:Because I am not interested in indexing these postXXX.html pages.

So are we don't worry.

I was advising you to let Google crawl postxx.html again only because the zero duplicate will HTTP 301 redirect these to the corresponding topic url, example : http://phpbb3.phpbb-seo.net/post59.html

So, given that postxx.html urls where already crawled by Google in your case, it's faster to redirect these, than to wait for Google to first really "believe" the robots.txt disallow and then drop them. This way, there would be no crawling rescheduling, only acknowledgement of the new location upon the next cache update.

In all cases, Google will discover and use your topic urls now that you installed the zero duplicate.

++