Google Gives Warnings (Webmasters)

[himanshu]

Hi I have restricted my URLs at robots.txt

[dcz]

it's normal and good to see ucp.php urls disallowed, it's related to :

[himanshu]

I read this note while installing (experimenting with the phpbb-seo)
since the site is ready I forgot this important thing.
Pls clarify me what to do know?

Remove all the instances from the robots.txt file right now
or after the site is 2 months old or the links or two months old???

[dcz]

The idea is simple.
If SEs already started to index the natural urls when you install a mod rewrite, you want to HTTP 301 redirect them to the new ones before you tell bots to discards them.
Because when you implement the robots.txt, you tell bot to discard immediately all the concerned urls (the phpBB3 natural ones in our case), which in our case, since the forum urls have changed, only concerns the url already indexed.
These already indexed urls are stored by bots in a crawl list, to schedule the future visits. So if you tell them to drop all of when they are still part of the crawl list, bots will do as if they where all disallowed (pretty close to deleted for them) and will pretty much start from scratch indexing your forum, like if no url where already indexed.
That's why it is wise to wait before you implement the robots.txt in such case.
This way bots will continue crawling using the already indexed urls from their crawling list, and will thus discover the HTTP 301 redirect and acknowledge the new urls to use for the same page.
Once these redirection are taken into account, at least two month after the redirecting started, you can safely implement the robots.txt, since you won't be disallowing urls from the bots crawling list (it will not contain any old natural urls after the redirecting will have occurred).

++

[himanshu]

thanks i did as you say.