logo
首页技术栈工具库讨论
cobweb

cobweb

Cobweb is a web crawler that can use resque to cluster crawls to quickly crawl extremely large sites which is much more performant than multi-threaded crawlers. It is also a standalone crawler that has a sophisticated statistics monitoring interface to monitor the progress of the crawls.
由 
bruceshi2021-01-13 收录
--
推荐
不推荐
更多信息
GitHub iconstewartmckee/cobweb225
RUBYGEMS
gem install cobweb
查看
标签
根据用户添加的标签生成
暂无标签