Controlling Web Robots

New article Search engines and other systems use robots or spiders to automatically crawl through your Web site. They parse the pages and store the data. It's possible to control where those robots go and what they do on your site using the robots.txt file.