How much important is to have robots.txt file?

Discussions about SEO Techniques implemented in your sites. Tests, Studies and results analysis.

Moderator: Moderators

How much important is to have robots.txt file?

Postby jiten702 » Fri Jun 25, 2010 10:54 am

Hi all......

As it is known to all that robots.txt file is use to specify how search engines should crawl your site's content.....but how much it is important to have it? Does it help in SEO?
jiten702
 
Posts: 10
Joined: Thu Jun 24, 2010 11:15 am

Advertisement

Re: How much important is to have robots.txt file?

Postby offsitenoc » Mon Aug 16, 2010 7:42 am

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (no HTML), that must be placed in your root directory, for example:
-http://www.musicmanias.0fees.net.com/robots.txt
offsitenoc
 
Posts: 23
Joined: Thu Aug 12, 2010 8:54 am

Re: How much important is to have robots.txt file?

Postby izzet37 » Fri Aug 20, 2010 9:31 pm

googlebot and other bots can avoid direct.
izzet37
 
Posts: 11
Joined: Tue Aug 17, 2010 8:57 pm

Re: How much important is to have robots.txt file?

Postby jacobkind » Thu Apr 05, 2012 8:36 am

Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from TheWebPole.com
jacobkind
 
Posts: 9
Joined: Wed Feb 08, 2012 9:40 am

Re: How much important is to have robots.txt file?

Postby Mulius » Thu Apr 05, 2012 10:56 am

It's not really important - as the others have said, you can for example disallow the search engines to visit part of your site. If you have to ask, it's probably not important.
Mulius
 
Posts: 11
Joined: Tue Mar 06, 2012 1:25 am


Return to SEO Techniques

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 6 guests