What is the robots.txt file?

[mackhunt88]

Please can anyone explain me what is robots.txt file and its correct format?

[justindepp2]

When the search engine crawlers to your website, it will look on your site a special file. The file named robots.txt to tell search engine spiders, your site is indexed Web pages and Web pages should be ignored.

[Nirav Shah]

Robts.txt file is a gateway for google crawler, it allows/disallows the content to be crawled by google.

[bryanbell]

Robots.txt is a text file which uploads on your website root folder to give instruction to search engine bots to which file they have to index and which are not

[ZacharyCampbell]

To begin we need to know what a web robot is, and is not. Thus, a Web robot is sometimes called spiders or web crawlers. You don't need a robots.txt file and most of the search engine robot crawlers will simply index your entire website if you don't have one.

[fdrix]

Well I totally agree with this saying that Robts.txt file is a gateway for google crawler, it allows/disallows the content to be crawled by google.

[sanfrancisco]

In simple words,Robots.txt is a file which tells crawler what to crawl and what not to crawl.It is very important for SEO process....

[mark44]

Its a file through which you can customize search engine spiders access for your site.

[connorbaron]

What is the purpose of robots.txt file in websites?
I want to know, what is the purpose of robots.txt file in websites?

How we can use it and can get benefits of it?

[thomasbell]

Robot.txt file is a set of instructions that tell seo robots which pages of your website to be listed and listed. In most cases, your website is consist of many files or files i.e. administration files, cgi-bin, image directory, which are not relevant to the google. Robot.txt helps tell robots what is useful and public for giving in google optimization robots and what is not.