A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading...

The robots. txt file is a simple text file (no html) that is placed in your website's root directory in order to tell the search engines.