So wouldn't that need to be unresctriced as that is vaulable content? Thier is several 100 URL's like that, that go to post.
Here is the robot text:
- Code: Select all
So would you recommend removing Disallow: /forum/post and/or Disallow: /forum/viewtopic.php or will that just cause google to index junk?
Also I might add it said "We can't currently access your home page because of a robots.txt restriction." but yet I am getting indexed and the line before that says this, " Googlebot has successfully accessed your home page. Last crawl date: Dec 31, 1969". Any ideas as to why, FYI my home page is Joomla CMS which links to phpBB forum. NOT interegrated, just linked to from my joomla to clear that up.