| :: |
| Author |
Message |
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
 Posted: Thu Jun 22, 2006 3:58 pm Post subject: where i must put my robots.txt file |
|
|
i want to ask some question about robots.txt file
imagine i have installed my forum in this address
-http://subdomain.mydomain.com
where i must put my robots.txt file ?
i've put one of my robots.txt file in WWW folder of my host
and another one in subdomain folder
the robots.txt file for root folder is like this
User-agent: *
Disallow: /subdomain/folder of page
and robots.txt file for subdomain is like this
User-agent: *
Disallow: /folder of page
is it correct? is it necessary to have two robots.txt file or one of them in root folder is sufficient ?
what a bout this domain
-http://www.mydomain.com/forum/
i must have two robots.txt file or one of them is enough
forum of my friend is installed in a folder in main domain
i put one robots.txt file in WWW folder and another one in forum
folder
in this two address
-http://www.siscenter.com/robots.txt
-http://www.siscenter.com/forum/robots.txt
is it correct ?
thanks |
_________________
چهار گوش - طراحی وب - مجله طراحی وب |
|
| Back to top |
|
 |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14327
|
| Posted: Thu Jun 22, 2006 4:15 pm Post subject: Re: where i must put my robots.txt file |
|
|
Well, the principle is simple, you want one robots.txt file per domain (including sub domain).
So if you installed for example sub.example.com and www.example.com in the same folders, eg :
www/ => main domain
www/sub/ => sub domain
then you want a robots.txt in both www/ and www/sub/
They shall both consider path starting from the domain's root.
So for the sud domain's robots.txt file you don't need to mention /sub/ in your disallows.
Then, no real need to disallow the sud/ folder in the www/ robots.txt, since you'd better use mod rewrite to redirect any access to the sud/ folder from the wwww.example.com main domain.
Hope I was clear 
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
| Posted: Fri Jun 23, 2006 9:42 am Post subject: Re: where i must put my robots.txt file |
|
|
my problem is this
i want to know
is this shape of URL a subdomain URL ?
| Code: | | http://www.domain.com/forum/ |
because i thought only this shape of URL is a subdomain
| Code: | | http://www.forum.domain.com/ |
and can you explain it a little ?
| Quote: | | Then, no real need to disallow the sud/ folder in the www/ robots.txt, since you'd better use mod rewrite to redirect any access to the sud/ folder from the wwww.example.com main domain. |
and
| DCZ wrote: | | Hope I was clear |
you are clear all the time
i have problem because my english is not good at this time
i need time to improve my english
thanks a lot |
_________________
چهار گوش - طراحی وب - مجله طراحی وب |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14327
|
| Posted: Fri Jun 23, 2006 9:58 am Post subject: Re: where i must put my robots.txt file |
|
|
| amir abbas wrote: | my problem is this
i want to know
is this shape of URL a subdomain URL ?
| Code: | | http://www.domain.com/forum/ |
because i thought only this shape of URL is a subdomain
| Code: | | http://www.forum.domain.com/ |
|
The second one only is the sub-domain URL, the first one is only here because in this example both domains are installed in the same directory.
The proper way to lock the sub-domain's files access from the main domain is :
| Code: | | RewriteRule ^forum/ http://forum.example.com/ [L,R=301] |
or :
| Code: | | RewriteRule ^forum/(.*) http://forum.example.com/$1 [L,R=301] |
The first one will only http 301 redirect www.example.com/forum/*.* to forum.example.com/ and the second one will keep URI as well : www.example.com/forum/*.* to forum.example.com/*.*
In both case, no need to mention the sub-domain's folder in the main robots.txt file. the forum/ folder is just not accessible any more from the main domain
This rule should be implemented in the main domain's .htaccess file, after the www redirection for example
| amir abbas wrote: |
| DCZ wrote: | | Hope I was clear |
you are clear all the time
i have problem because my english is not good at this time
i need time to improve my english
thanks a lot |
With phpBB SEO, all of us, because I am part of it  , speaking English like Spanish cows will improve our English vocabulary while Search Engine Optimizing our web-sites
++ |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
AmirAbbas
phpBB SEO Team
Joined: 11 May 2006
Posts: 529
Location: IRAN
|
| Posted: Fri Jun 23, 2006 10:12 am Post subject: Re: where i must put my robots.txt file |
|
|
according to your texts
a URL with this form
| Code: | | http://www.domain.com/forum/ |
without this rewrite rule
| Code: | | RewriteRule ^forum/(.*) http://forum.example.com/$1 [L,R=301] |
is not a subdomain
i think i made a big problem for my friend's forum
i put two robots.txt file one of them in main domain and another one in
forum/ folder. after 3 week only 100 pages of his forum is indexed in
google that all of that page was indexed before i put robots.txt file on forum/ folder  |
_________________
چهار گوش - طراحی وب - مجله طراحی وب |
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14327
|
| Posted: Fri Jun 23, 2006 10:24 am Post subject: Re: where i must put my robots.txt file |
|
|
| amir abbas wrote: | according to your texts
a URL with this form
| Code: | | http://www.domain.com/forum/ |
without this rewrite rule
| Code: | | RewriteRule ^forum/(.*) http://forum.example.com/$1 [L,R=301] |
is not a subdomain
|
Even without actually
If those two, the main domain and the sud-domain where installed independently, there would not be any way to access one's files from the other.
Here it's possible just because both are installed in the same physical directory (www/). This kind of setup is convenient to be able to includes files form forum/ in www/ but both path are fully installed.
| amir abbas wrote: |
i think i made a big problem for my friend's forum
i put two robots.txt file one of them in main domain and another one in
forum/ folder. after 3 week only 100 pages of his forum is indexed in
google that all of that page was indexed before i put robots.txt file on forum/ folder |
I don't understand sorry.
Because you still should put one robots.txt per domain which here leads us to put on in www/ , for the main domain, and the second in forum/, for the sub domain.
So it all depends on their content, but they should be there. I only mentioned that the main robots.txt located in www/ should not mention the forum/ folder since we are using a nice redirection here.
This could explain why few pages are indexed for now, because bots could have loaded the robots.txt before they tested URLs, and thus, not trying and got rid of the URLs mentioning forum/ before they'd tried them and eventually got redirected.
In all cases, don't worry, mx Google sitemaps is watching our backs and will have Google indexing all pages, 3 week is just not this much, one month and a half should be when we start worrying |
_________________
Useful links :
SEO Forum || SEO Directory || SEO phpBB || SEO phpBB3 || Search
____________________
Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Référencement phpBB3 || Recherche |
|
| Back to top |
|
|
Hannibal_King
PR0
Joined: 22 Jun 2006
Posts: 78
|
| Posted: Sat Aug 19, 2006 9:05 pm Post subject: Re: where i must put my robots.txt file |
|
|
i have a question to this...
if i have my forum on ftp in www directory, is this robots.txt good?
| Code: | User-agent: *
Disallow: /admin
Disallow: /attach_mod
Disallow: /cache
Disallow: /db
...
.. |
|
|
|
| Back to top |
|
|
dcz
Administrateur - Site Admin
Joined: 28 Apr 2006
Posts: 14327
|
|
| Back to top |
|
|
|
|
