How do I choose robots dir?

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Moderator: Moderators

How do I choose robots dir?

Postby h4ck3r » Fri Jun 19, 2009 5:22 am

I installed my forum on -www.site.com/forum/ . So it gets to root's robots (-www.site.com/robots.txt)
How I can change?
h4ck3r
 
Posts: 9
Joined: Sat Jun 06, 2009 4:19 am

Advertisement

Re: How do I choose robots dir?

Postby dcz » Fri Jun 19, 2009 7:37 am

Change what ?
There is only one location for robots.txt, and it's the domain's root.
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21219
Joined: Fri Apr 28, 2006 9:03 pm

Re: How do I choose robots dir?

Postby marteen147 » Fri Jul 09, 2010 8:43 pm

You can easily make this by yourself and this is how to do it.

Open your notepad and save it in your main directory as robots.txt. Some malware search engines are used to find weaknesses in your website or to find your E-mail adress so that they can send you spam E-mails. In these cases you can block them out by using the robots.txt. The first thing the majority of search engines will do before searching through your directories is to seek a robots.txt file. If they do not find one they will then go through your index page and move on throughout your website. Here are a few examples of what you can do.

User-agent: *
Disallow: /

By writing the preceding in robots.txt file you will disallow ALL search engines to search your site. In this manner your entire site is locked and the search engines have no access to it. The star that you insert for User-agent means that all search engines are concerned and subject to what you disallow them to view. Something else you can do is the complete opposite.

User-agent: *
Disallow:

By putting nothing for Disallow you are giving access to all search engines, along with the permission to search and verify your entire website. You can also choose to specify what search engines are subject to these instructions as the following example demonstrates.

User-agent: Google
Disallow:
Last edited by SeO on Sun Nov 21, 2010 8:00 pm, edited 1 time in total.
Reason: Links removed (rule 5a : Linking offsite is allowed if links are being offered in a "relevant discussion context")
marteen147
 
Posts: 1
Joined: Fri Jul 09, 2010 8:31 pm

Re: How do I choose robots dir?

Postby thezodiac » Mon Sep 20, 2010 4:21 pm

The robots.txt file should be located in the root of your domain.
thezodiac
 
Posts: 34
Joined: Mon Sep 20, 2010 10:43 am


Return to roBots

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 2 guests