Your website directories will be crawled and indexed on search engine according to robots.txt instructions. A full access or a full restriction or customized access or restriction can be imposed through robots.txt. robots.txt contains the search bots or others bots name, directory list allowed or disallowed to be indexing and crawling for bots, time delay for bots to crawl and indexing and even the sitemap url. Robots.txt is text file that reside on website root directory and contains the instruction for various robots (mainly search engine robots) for how to crawl and indexing your website for their webpage.
