robots.txt
robots.txt is a text file that tells crawlers which URLs on a site they may access. What it can and can't do, where it must live and why it can't hide a page.
robots.txt is a plain text file that tells search engine crawlers which URLs on a site they are allowed to access. It sits at the root of a host, for example example.com/robots.txt, and contains rules for all crawlers or for specific ones, such as Googlebot. It can also list the address of your sitemap.
Google says robots.txt is used mainly to avoid overloading a site with requests: "it is not a mechanism for keeping a web page out of Google." A page blocked in robots.txt can still appear in results if other pages link to it, just without Google having read it. To keep a page out of search, use a noindex rule or put the page behind a password. And don't block that page in robots.txt at the same time, because Google can't see a noindex rule on a page it isn't allowed to crawl.
The file applies only to the host, protocol and port it is served on. Google's specification says a robots.txt on a subdomain is valid only for that subdomain, so blog.example.com needs its own file. Google also ignores content beyond the first 500 KiB of the file.