Robots.Txt

There are plenty of places on the web that have detailed descriptions of how to correctly implement robots.txt files on your sites.

Here are just a few of the more important ones that I have found..

Database of Web Robots ( http://www.robotstxt.org/wc/active/html/index.html )

The WebRobots FAQ ( http://www.robotstxt.org/wc/faq.html )

The Web Robots Page ( http://www.robotstxt.org/wc/robots.html )

Robots.txt Checker ( http://tool.motoricerca.info/robots-checker.phtml )

SearchTools.com (http://www.searchtools.com/robots/robots-txt.html

Robots.txt File Generator ( http://www.webtoolcentral.com/webmaster/tools/robots_txt_file_generator/ )

Robots Exclusion Standard ( http://en.wikipedia.org/wiki/Robots.txt )