Robots.txt File vs Robots meta tag vs X-Robots-Tag
The Robots.txt File, Robots meta tag and X-Robots-Tag are used to control or communicate what search engine spiders do on your website or web application.
What is Robots.txt File? What are the Different types of bots or Web Crawlers?
Robots.txt is a standard text file is used for websites to communicate with web crawlers or bots. It is used for the purpose of Web indexing or spidering.