Before we go solution steps, we need to understand what the Robots.txt file is?
Robots.txt is a pure text file that asks search engine what URLs or contents are supposed to be crawled. While publishing a new page or posting some content to your website, search engine bots can crawl this material to index it to the search results. At the same time, If you do have certain parts of your website that you don’t want to be indexed, you can tell search bots to skip them so that they don’t show on the search results page.
Let’s go back to our solution for Prestshop 1.7.x. when you see the errors in your Google search console like “URL blocked by robots.txt” and “URL is not on Google: indexing errors”. Attached screenshot is a reference.
This kind of error will happen sometimes when you do the Prestashop version upgrade, migration or any other. Please don’t worry and just follow the below steps.
1/. type www.yourdomain.com/robots.txt to verify whether you have this file or not?
2/. Make sure the following lines are available in your Robots.txt files
3/. You can use the below link to check your Robots.
4/. If your domain is not appear in your property selection, you can use the following url to add it
5/. You also need to check if you can find any sitemap files like below in your backend. It is better to remove all the sitemap files first. We will re-generate it.
1/. The following we will re-generate Robots.txt.
2/. Log into your Prestashop admin portal. Go to “Configure” section, under submenu “Shop Parameters” and “Traffic & SEO”, scroll down to bottom and click “Generate robots.txt file”
3/. https://www.google.com/webmasters/tools/robots-testing-tool?pli=1 select your domain property
4/. In robots.txt Tester page, scroll down to the bottom and put “robots.txt” in the Enter a URL to test if it is blocked field. Then click “Test” red button to release the updated version.
5/. You will see “Latest version seen on date and time OK (200) xxxx Bytes”
1/. Regaring how to generate a sitemap, you may refer to the below url.
** Attention: Make a full backup prior to the change