i want to ask some question about robots.txt file

imagine i have installed my forum in this address

-http://subdomain.mydomain.com

where i must put my robots.txt file ?

i've put one of my robots.txt file in WWW folder of my host
and another one in subdomain folder

the robots.txt file for root folder is like this

User-agent: *
Disallow: /subdomain/folder of page

and robots.txt file for subdomain is like this

User-agent: *
Disallow: /folder of page


is it correct? is it necessary to have two robots.txt file or one of them in root folder is sufficient ?

what a bout this domain

-http://www.mydomain.com/forum/

i must have two robots.txt file or one of them is enough
forum of my friend is installed in a folder in main domain
i put one robots.txt file in WWW folder and another one in forum
folder

in this two address

-http://www.siscenter.com/robots.txt
-http://www.siscenter.com/forum/robots.txt

is it correct ?


thanks

Well, the principle is simple, you want one robots.txt file per domain (including sub domain).

So if you installed for example sub.example.com and www.example.com in the same folders, eg :

www/ => main domain
www/sub/ => sub domain

then you want a robots.txt in both www/ and www/sub/

They shall both consider path starting from the domain's root.

So for the sud domain's robots.txt file you don't need to mention /sub/ in your disallows.

Then, no real need to disallow the sud/ folder in the www/ robots.txt, since you'd better use mod rewrite to redirect any access to the sud/ folder from the wwww.example.com main domain.

Hope I was clear

++

my problem is this

i want to know

is this shape of URL a subdomain URL ?

Code: Select all

http://www.domain.com/forum/

because i thought only this shape of URL is a subdomain

Code: Select all

http://www.forum.domain.com/

and can you explain it a little ?

Then, no real need to disallow the sud/ folder in the www/ robots.txt, since you'd better use mod rewrite to redirect any access to the sud/ folder from the wwww.example.com main domain.

and

DCZ wrote:Hope I was clear

you are clear all the time
i have problem because my english is not good at this time
i need time to improve my english

thanks a lot

amir abbas wrote:my problem is this

i want to know

is this shape of URL a subdomain URL ?

Code: Select all

http://www.domain.com/forum/

because i thought only this shape of URL is a subdomain

Code: Select all

http://www.forum.domain.com/

The second one only is the sub-domain URL, the first one is only here because in this example both domains are installed in the same directory.

The proper way to lock the sub-domain's files access from the main domain is :

Code: Select all

RewriteRule ^forum/ http://forum.example.com/ [L,R=301]

or :

Code: Select all

RewriteRule ^forum/(.*) http://forum.example.com/$1 [L,R=301]

The first one will only http 301 redirect www.example.com/forum/*.* to forum.example.com/ and the second one will keep URI as well : www.example.com/forum/*.* to forum.example.com/*.*

In both case, no need to mention the sub-domain's folder in the main robots.txt file. the forum/ folder is just not accessible any more from the main domain

This rule should be implemented in the main domain's .htaccess file, after the www redirection for example

amir abbas wrote:

DCZ wrote:Hope I was clear

you are clear all the time
i have problem because my english is not good at this time
i need time to improve my english

thanks a lot

With phpBB SEO, all of us, because I am part of it , speaking English like Spanish cows will improve our English vocabulary while Search Engine Optimizing our web-sites

++

according to your texts

a URL with this form

Code: Select all

http://www.domain.com/forum/

without this rewrite rule

Code: Select all

RewriteRule ^forum/(.*) http://forum.example.com/$1 [L,R=301]

is not a subdomain

i think i made a big problem for my friend's forum
i put two robots.txt file one of them in main domain and another one in
forum/ folder. after 3 week only 100 pages of his forum is indexed in
google that all of that page was indexed before i put robots.txt file on forum/ folder

amir abbas wrote:according to your texts

a URL with this form

Code: Select all

http://www.domain.com/forum/

without this rewrite rule

Code: Select all

RewriteRule ^forum/(.*) http://forum.example.com/$1 [L,R=301]

is not a subdomain

Even without actually

If those two, the main domain and the sud-domain where installed independently, there would not be any way to access one's files from the other.
Here it's possible just because both are installed in the same physical directory (www/). This kind of setup is convenient to be able to includes files form forum/ in www/ but both path are fully installed.

amir abbas wrote:i think i made a big problem for my friend's forum
i put two robots.txt file one of them in main domain and another one in
forum/ folder. after 3 week only 100 pages of his forum is indexed in
google that all of that page was indexed before i put robots.txt file on forum/ folder

I don't understand sorry.

Because you still should put one robots.txt per domain which here leads us to put on in www/ , for the main domain, and the second in forum/, for the sub domain.

So it all depends on their content, but they should be there. I only mentioned that the main robots.txt located in www/ should not mention the forum/ folder since we are using a nice redirection here.

This could explain why few pages are indexed for now, because bots could have loaded the robots.txt before they tested URLs, and thus, not trying and got rid of the URLs mentioning forum/ before they'd tried them and eventually got redirected.

In all cases, don't worry, mx Google sitemaps is watching our backs and will have Google indexing all pages, 3 week is just not this much, one month and a half should be when we start worrying

i have a question to this...
if i have my forum on ftp in www directory, is this robots.txt good?

Code: Select all

User-agent: *
Disallow: /admin
Disallow: /attach_mod
Disallow: /cache
Disallow: /db
...
..

Yes if your forum is installed at his domain root (meaning www.example.com/ followed by no path) your paths are correct in your robots.txt, and it should always be uploaded ate the domain's root as well.

check the release thread of the mod rewrite you are using, there is a (strongly) suggested robots.txt for each.

++