How to edit WordPress robots.txt file for better SEO but first learn What is WordPress robots.txt file and why you need one. It is a file which tells Google search engine bot what to show and hide in search results and indexing.  There are few misunderstanding related to indexing and non-indexing of content in Robots.txt. I am not going make a long article just follow simple steps, and that’s all.



how to edit wordpress robots.txt file

Why You should Edit WordPress robots.txt file

It is an essential part of SEO, and It is important because some time search engine crawls root directory and WordPress admin pages too which is a security risk for any blog. Maybe your blog has the robots.txt file but it is miss-configured, or you are not aware how to optimize it.

Cannot find robots.txt in WordPress?
Robots.txt file normally located in the root folder of your hosting Cpanel account. You can go to the file manager and there you will find your robots.txt file.

robots.txt file location

WordPress robots.txt disallow

User-agent: Googlebot

Disallow: /cgi-bin

These two lines will allow Google bot to index every page of your blog accept cgi-bin folder which is the root directory folder. It means bots will not index cgi-bin folder

WordPress robots.txt allow

User-agent: Googlebot

Allow: /cgi-bin

Simple if you use ALLOW command defiantly google bot will index this folder, and this is the way how to use ALLOW and DISALLOW commands in the robots.txt file.

Don’t include Sitemap in WordPress robots.txt

A sitemap is also the most important part of any blog because it has everything like pages, categories,  post, etc. In my opinion, Sitemap should not include in the robots.txt file.
WordPress Robots.txt file Example

Parts of Google Bots

How to Edit WordPress robots.txt File

Here is an example of my robots.txt file and I hope you can copy it and use it on your blog. All you need to do is create a new text file on your computer using notepad or any text editor and save it as robots.txt. Next, simply upload it to your site’s root folder. It is easy to create WordPress robots.txt File but be careful in editing.

How  to check  WordPress robots.txt file

To check your robots.txt file simply type   in your browser and your robots.txt file will be displayed.  Please replace your domain name with your sites name.

Check WordPress robots.txt file in google webmaster tools

This is the last and important point because we are dealing with Google search engine bot so we must go to our google webmaster tools and choose robots.txt from Crawl menu. There you can see tester option submit it and you will see the results.

robots.txt file testing

After this article, I hope now you will not ask this question how to edit WordPress robots.txt file because you can make and configure your WordPress Robots.txt File and optimize it as per your needs. If you have any questions, please comment below.

How To Edit WordPress Robots.txt File For Better SEO
  • How To Edit WordPress Robots.txt File For Better SEO


Best way to configure robots.txt file