Discovered Pages
[Discover] How to write a multi-threaded webcrawler in Java http://www.few.vu.nl/~andreas/programming/webcrawler/index.html
(research opensource webcrawler design) 
[Discover] WebCrawler Web Search http://www.webcrawler.com/webcrawler300/ws/index/_iceUrlFlag=11?_IceUrl=true (search www)
[Discover] WebCrawler Web Search http://www.webcrawler.com/ (search)
[Discover] The HarvestMan WebCrawler http://www.harvestmanontheweb.com/ (python crawler copy opensource)
[Discover] DataparkSearch Engine - open source search engine http://www.dataparksearch.org/ (search crawler searchengine spider)
[Discover] Yahoo!'s Web Crawler http://help.yahoo.com/l/us/yahoo/search/webcrawler/index.html (search crawler seo slurp)