Do i still need to make an robots.txt

[emielvenlo]

On my forum: [MOD]link removed[/MOD] I have made the following mods:
Advanced mod_rewrite
zero dupe
optimal titles

Do i still need to make an robots.txt for instance to not let google index posts(dupe's from topic) and members??

[HB]

The zero dupe will redirect viewtopic.php / viewforum.php to the new URLs; you don't want to block search engine access. Below are robots.txt rules I recommend:

Code: Select all

User-agent: *
Disallow: /forums/groupcp.php
Disallow: /forums/memberlist.php
Disallow: /forums/member
Disallow: /forums/faq.php
Disallow: /forums/login.php
Disallow: /forums/modcp.php
Disallow: /forums/posting.php
Disallow: /forums/search.php
Disallow: /forums/profile.php
Disallow: /forums/privmsg.php
Disallow: /forums/viewonline.php

dcz's list is slightly different:

http://www.phpbb-seo.com/robots.txt

They're essentially the same except for index.php, viewtopic.php, and viewforum.php. I omitted them to allow the search engines a chance to be redirected.

[dcz]

Yes, you should not disallow :

Code: Select all

Disallow: /boards/viewtopic.php
Disallow: /boards/viewforum.php
Disallow: /boards/index.php

when migrating urls, but this can help out once it's done. As these will prevent all the advanced pagination sorting options to be cached as well.

So while migrating urls, I'd say, get rid of these for couple month, and put them back after, when bots already stopped using them.

++

[emielvenlo]

Yes, you should not disallow :

Code: Select all

Disallow: /boards/viewtopic.php
Disallow: /boards/viewforum.php
Disallow: /boards/index.php

when migrating urls, but this can help out once it's done. As these will prevent all the advanced pagination sorting options to be cached as well.

So while migrating urls, I'd say, get rid of these for couple month, and put them back after, when bots already stopped using them.

++

In this case its a new forum (not a month old yet), so i dont think google has much indexed yet. Is it better to keep the rules if its a new forum because its not really migrating?
(this is not brandpreventieforum, but the link was removed)

[dcz]

yes, new install is not like migrating, you can tell bot to ignore these, even though they are redirected

++