airforce1 wrote:robots.txt is OK for guiding search engine crawlers behaviors but I think it might be a file could be used by hackers.
dcz wrote:The robots.txt is just a text file with instructions that search engine crawlers may use for themselves.
The typical usage is to disallow some urls, for example :
- Code: Select all
Users browsing this forum: Baidu [Spider] and 6 guests