How much important is to have robots.txt file?

Discussions about SEO Techniques implemented in your sites. Tests, Studies and results analysis.

Moderator: Moderators

How much important is to have robots.txt file?

Postby jiten702 » Fri Jun 25, 2010 10:54 am

Hi all......

As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?
jiten702
 
Posts: 10
Joined: Thu Jun 24, 2010 11:15 am

Advertisement

Re: How much important is to have robots.txt file?

Postby offsitenoc » Mon Aug 16, 2010 7:42 am

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:
-http://www.musicmanias.0fees.net.com/robots.txt
offsitenoc
 
Posts: 23
Joined: Thu Aug 12, 2010 8:54 am

Re: How much important is to have robots.txt file?

Postby izzet37 » Fri Aug 20, 2010 9:31 pm

googlebot and other bots can avoid direct.
izzet37
 
Posts: 11
Joined: Tue Aug 17, 2010 8:57 pm


Return to SEO Techniques

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 3 guests