Ideal robots.txt with the zero dup

j2ross · Joined: 21 May 2008 Posts: 12

Ok so I read about the correct robots file to use when installing the advanced mod rewrite, but wasn't clear about what to use with zero dup.

Since all my existing viewtopic.php and viewforum.php will be getting a 301 to the newly rewritten pages is it safe to say that I should not be disallowing those urls in the robots file?

SeO

Well, you should not disallow the redirected files in your robots.txt only if they previously where indexed, otherwise, or once the redirecting is acknowledged by SEs, the advised robots.txt is the same with and without the zero dupe. With the zero dupe on, it's of course not really important, but still, it can make things clearer for SEs.