Making WordPress and Robots.txt work for you
Every website should have a robots.txt file! But, unfortunately in the rush to get a new website launched, this unseen file seens to get forgeooten and most people simply forget all about their unseen robots.txt file .
When using wordpress, and trying to minimize your duplicate content exposure to the search engines, then having a proper robots.txt file is really important. I blogged about this a few months ago, and you can find the post here.
So a good wordpress robots.txt file should look something like this, including the link to your sitemap (in XML format) at the bottom.
User-agent: *
Disallow: /wp-content/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /wp-includes/
Disallow: /trackback/
Disallow: /cgi-bin/
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /wp-
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /page/
Disallow: /date/
Disallow: /category/
Disallow: /archive/
Disallow: /rss/
# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*
sitemap:http://www.yoursite.com/sitemap.xml
But, like the true geek I really am. I got tired of typing this in every time I made a new site. So I created a wordpress plugin that makes this plugin for you.
Yes, there are other wordpress plugin out there, some better, some worse, but this one works exactly like I want it to work. In otherwords, it optimized to the MAX!
So if you want to get the plugin for free, simply download it from here.
Go to your plugin menu in the admin area, and activate.
Then click on the BH Robots.txt menu item in the Settings sections and DON’T forget to remove the leading # symbole at the start of the last line. Only remove it if you have a sitemap, otherwise leave it there.
Let me know what you think about this plugin.
