what is robots.txt

Introduction ‘robot.txt’ is a small text file & it plays an important role in technical SEO and website crawling instructions to automated clients, commonly called bots, crawlers, or spiders. A robots.txt file tells compliant search engine crawlers which URLs they are allowed or not allowed to request from a website. It is mainly used to