Robot .txt is a file in the website which allows the search engines to crawls the website URLs and index process, this file will navigate the bots while crawling the website and provide instructions to allow which page to crawl and indexed. On your web page, you have the content needed for your website but no need for search rankings.

Step-1

Enter into the website checkseo.on a page with a search bar will appear on the screen with a review button on the side. Now enter your URL in the search box click for review

Step-2

A score page will appear on the screen, on-page site score helps to improve your search results

Step-3

In your website, if robot.txt file is missing search engines cannot understand which page to crawl and the content of your website and its effects on the search rankings of the website. the robot.txt file allows the bots and navigates which URL to crawl and indexed. If your website doesn’t have robot.txt file then the report will be displayed as the image given below

Step-4

Robot.txt file provides a direction to the bots on which page to crawl and indexing for search rankings in all search engines. This allows the bots which page to allow and disallow for indexing in search results. In most of the cases in the web page content is required for the website but no need for indexing in search results, robot.txt file allows to crawl the required pages for indexing. The report will be displayed in the above image if your website has robot.txt files