How robots.txt works?

The robots.txt is just a text file with instructions that search engine crawlers may use for themselves.
The typical usage is to disallow some urls, for example :

Code: Select all

User-agent: *
Disallow: /file.php

Will ask SE not to index example.com/file.php. User-agent: * means all user agents.
Few other things could be done with robots.txt, but since SE do only follow these directives on a voluntary basis (major ones do or tend to, but some others do not at all), these should only be considered as hints, not as rules you could actually rely on.

++

robots.txt is OK for guiding search engine crawlers behaviors but I think it might be a file could be used by hackers.

Used for what ?

I mean, of course it's a bit stupid if you for example disallow your back-office directory while it is password protected, because it's a way to tell the world its name, but it's not really going to help an hacker a lot. At least a hacker should not wait after this kind of hints to sneak in.

++

It's to maintain for 301 and 404 error.

robot.txt file is mainly use to let the crawler now what to crawl and what not...when the search crawler comes to any site it searches the robot.txt and crawl the site accordingly.

airforce1 wrote:robots.txt is OK for guiding search engine crawlers behaviors but I think it might be a file could be used by hackers.

Its main use now is to tell search engine spiders not to crawl certain pages or files on your site. It is only a txt file and as such is not an executable file so I cannot see that it would be of any interest to hackers.

robots.txt is a text file which is specially used for web page.Google spider crawl over a site for little bit of sec,so within this period it is not possible to crawl all the pages of your site,so xml-site map is necessary for it. if we want spider is not crawl one page (the page which is not necessary)then in that case we can stop the spider not to crawl that page by using robots.txt file.

The concept of robots.txt is this: a robot wants to visit the site -http://www.musicmanias.0fees.net/. Before it does anything, it first looks for -http://www.musicmanias.0fees.net/robots.txt, to find out which pages it can index or not. If it can’t find the filename, it will go ahead and index everything on that directory.

dcz wrote:The robots.txt is just a text file with instructions that search engine crawlers may use for themselves.
The typical usage is to disallow some urls, for example :

Code: Select all

User-agent: *
Disallow: /file.php

thank u dcz.. it was useful 4 me

By robots.txt we tell to Google that which page is allow for crawling or which not....

In a nutshell: Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
http://www.robotstxt.org/robotstxt.html