- Controlling how search engines access and index your website
- The Robots Exclusion Protocol
- Robots Exclusion Protocol: now with even more flexibility
Danny Sullivan has several write ups on robots.txt and Copiepresse as well.
- Search Engines, Permissions & Moving Forward In Copyright Battles – Daggle.com
- Google’s Belgium Fight: Show Me The Money, Not The Opt-Out, Say Publishers – Search Engine Watch