robots.txt

Short answer

robots.txt is a file at a site's root telling crawlers which paths they may or may not crawl. It is used to keep bots away from low-value or private sections and to manage crawl load. Blocking a URL in robots.txt does not guarantee it stays out of search results; noindex handles that.