craig> web>
     robots
:
Web Robots and Spidering
1.
   Robots Exclusion Protocol [specification]
This guide is aimed at Web Server Administrators who want to use the Robots Exclusion Protocol.
URL: http://info.webcrawler.com/mak/projects/robots/exclusion-admin.html
2.
   Web Stripper
Some sites can be difficult to grab if they come from CGI scripts or contain JavaScript. WebStripper can't promise to download 100% of websites but it in most cases it does a very good job of downloading them.

URL: http://www.webstripper.net/

  Jump:
  Search:
toolbar help Random Link   Map   Flat Listing   Brief Listing   Login