Robots are programs which go off and can trawl through websites and all documents that are referenced within. Most people know them for their indexing service like the Googlebot, but they also have a few other functions including:

• Indexing
• HTML/Link Validation
• "What's new" monitoring
• Mirroring

To know more about these programs and how to control their visit through your site check out http://www.robotstxt.org/wc/robots.html