Tag Archives: robots.txt

Every website should have a robots.txt file! But, unfortunately in the rush to get a new website launched, this unseen file seens to get forgeooten and most people simply forget all about their unseen robots.txt file .

When using wordpress, and trying to minimize your duplicate content exposure to the search engines, then having a proper robots.txt file is really important. I blogged about this a few months ago, and you can find the post here. Continue reading →