How To Change And Use Robots.txt In Wordpress Blog

Frankly many of us are not even aware that there is something called robots.txt and what it does and what is the importance of using robots.txt in your blogs and websites.

Infact its not a compulsion that you need to have a robots.txt in your blog, in fact you don’t need one but still if you want to pass on any special instructions to search engine spiders then robots.txt is a must have file on your server, in this article we will see how you can effectively utilize this text file for your wordpress blogs when you want to pass on any special instructions to search engines.

First thing a search engine spider looks for when it reaches a site or blog is robots.txt file to see if there is any special instructions, instructions may be like you may not want search engines to crawl some parts of your blogs or some sections of your blog and these kind of instructions can be passed to search engine spiders with this file. But mostly you won’t be able to locate this file through your ftp program, so if you want to access your robots.txt file from your admin panel and alter it then you must use this plugin called KB Robots also this plugin is of any use only if your blog is hosted in the root of your server not to some other directories like say www.yourdamin.com/blog if that is the case create your custom robots.txt file in your machine and upload it to the root of your server most cases under public_html folder

Wordpress comes with a default robots.txt file that looks like this
User-agent: *
Disallow:
What does this mean? it means allow all the search engine bots to crawl through your entire blog without any exception.

To ban all the search engine bots from crawling through your entire blog you can use this code.
User-agent: *
Disallow: /

This will ban all the search engine bots from crawling any parts of your blog and will stay away from your blog which of course we don’t want.

You can object specific bots from crawling your webpages like Googlebot or Msn bot then you can pass that instruction to that specific bot like this.
User-agent: Googlebot
Disallow: /

In the code above we have asked Googlebot to stay away from the site, you can find the list of robots here

To allow only specific bots and ban others then you can do it like this.

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

To disallow specific sections of your blog from being crawled then you can pass the instruction by using the code like this.

User-agent: *
Disallow: /archives/

you can substitute the archives with whatever you want, if you don’t want specific category not to be crawled then replace it with that category name and that category will not be indexed and crawled by search engines.

This is a copy of my robots.txt you can access it at http://seoyourblog.com/robots.txt

you can see even google uses a robots.txt file http://google.com/robots.txt almost all the websites on the internet uses robots.txt file and you can access this file by adding a robots.txt at the end of their domain.

Tags: changing robots.txt, wordpress robots.txt

Related posts:

1. How To Prevent Content Theft From Your Wordpress Blog
2. 4 Steps To Understand How Search Engines Read Websites
3. Speed up Your Wordpress Blog With Two Simple Tricks
4. How To Change User Roles In Wordpress
5. How To Secure Your Wordpress Blog From Hackers

Tagged as: changing robots.txt, wordpress robots.txt