What is the robots.txt file?

[janel]

Robots.txt file is in the root directory of a website created to direct the activity of search engine crawlers or spiders

[edwardleon]

A robots.txt file is a text file in a simple format which gives information to web robots such as search engine spiders about which parts of your website they are and aren't allowed to visit.

[Elroy]

If you want that some files of your site not to be crawled then you can use robot.txt file .....