my robots look like this
User-agent: *
Disallow: /phpBB2/viewtopic.php
Disallow: /phpBB2/viewforum.php
Disallow: /phpBB2/index.php?
Disallow: /phpBB2/posting.php
Disallow: /phpBB2/groupcp.php
Disallow: /phpBB2/profile.php
Disallow: /phpBB2/memberlist.php
Disallow: /phpBB2/search.php
Disallow: /phpBB2/login.php
Disallow: /phpBB2/faq.php
Disallow: /phpBB2/about
Disallow: /phpBB2/member
Disallow: /phpBB2/post
Disallow: /phpBB2/mark
Disallow: /phpBB2/updates-topic.html
Disallow: /phpBB2/stop-updates-topic.html
Disallow: /phpBB2/ptopic
Disallow: /phpBB2/ntopic
Disallow: /phpBB2/album_rate.php
Disallow: /phpBB2/privmsg.php
I have since added some pages like
http://www.asia-expatsforum.com/phpBB2/ ... -links.php
are these new pages being spidered ? because I got a message from one of those auto link checkers that vietnam-links.php was blocked by robots .text

English |
French

