I need Robots.txt to block spiders from root directory

daddyo · Joined: 12 Sep 2007 Posts: 38

Currently, my robots file is this:

User-agent: *
Disallow: /phpBB/posting.php
Disallow: /phpBB/groupcp.php
Disallow: /phpBB/search.php
Disallow: /phpBB/login.php
Disallow: /phpBB/privmsg.php
Disallow: /phpBB/post
Disallow: /phpBB/member
Disallow: /phpBB/profile.php
Disallow: /phpBB/memberlist.php
Disallow: /phpBB/faq.php

I have this in my root web folder. I would also like to block spiders from crawling in my root folder as well, but since I'm unfamiliar with the syntax, I don't want to create somewthing that will block my forum URLs as well.

How do I use Disallow to disallow access to my root folder, but still keep the same rules as above?

dcz

The problem is, if you really disallow the root folder, you'll disallow everything.

A disallow on a folder will be active on all pages in the folder, it's the same for the root folder.

If you want to disallow a type of page in the root folder, you can use :

daddyo · Joined: 12 Sep 2007 Posts: 38

Thanks.

The only problem is that in order to get individual pages to not be indexed, I have to list them one-by-one in the robots.txt, thus exposing my files to anyone looking at my robots.txt file.

I guess I need to put them in their own subdirectory.

dcz

The sub directory is a good idea.

You can as well ad the noindex tag in the meta tags. It's pretty much the same effect as the robots.txt, but performed on individual pages.

something like :

daddyo · Joined: 12 Sep 2007 Posts: 38

Thanks.

I think for now I'll try the meta tag approach, since I have lots of scripts that depends on other scripts and it might take too long to track down dependencies if I moved them all to a subfolder.