If you have a well designed and well optimized website with keyword rich content to attract visitors and search engines that is really great but you are missing something that is very important. Do you know what that is? That is Robots.txt file.

Robots.txt file has lots of importance as it allows spiders or crawlers to allow or disallow to crawl all pages of a website or a particular webpage. Sometimes people have some confidential data on their website and by using robots.txt file they can restrict crawlers or spider to not crawl or index that particular page so no one can reach on that page and in this way confidential data on that page will be secure.

More: continued here