function of the Robots.txt file?

It provides instructions to robots which pages and directories not to index.