by marteen147 » Fri Jul 09, 2010 8:43 pm
You can easily make this by yourself and this is how to do it.
Open your notepad and save it in your main directory as robots.txt. Some malware search engines are used to find weaknesses in your website or to find your E-mail adress so that they can send you spam E-mails. In these cases you can block them out by using the robots.txt. The first thing the majority of search engines will do before searching through your directories is to seek a robots.txt file. If they do not find one they will then go through your index page and move on throughout your website. Here are a few examples of what you can do.
User-agent: *
Disallow: /
By writing the preceding in robots.txt file you will disallow ALL search engines to search your site. In this manner your entire site is locked and the search engines have no access to it. The star that you insert for User-agent means that all search engines are concerned and subject to what you disallow them to view. Something else you can do is the complete opposite.
User-agent: *
Disallow:
By putting nothing for Disallow you are giving access to all search engines, along with the permission to search and verify your entire website. You can also choose to specify what search engines are subject to these instructions as the following example demonstrates.
User-agent: Google
Disallow:
Last edited by
SeO on Sun Nov 21, 2010 8:00 pm, edited 1 time in total.
Reason: Links removed (rule 5a : Linking offsite is allowed if links are being offered in a "relevant discussion context")