标签 awl 下的文章

nutch源码笔记


主要类分析:

一、 org.apache.nutch.crawl.Injector:
1,注入url.txt
2,url标准化
3,拦截url,进行正则校验(regex-urlfilter.txt)
4,对符URL标准的url进行map对构造<url, CrawlDatum>,在构造过程中给CrawlDatum初始化得分,分数可影响url host的搜索排序,和采集优先级!
5,reduce只做一件事,判断url是不是在crawldb中已经存在,如果存在则直接读取原来CrawlDatum,如果是新host,则把相应状态存储到里边(STATUS_DB_UNFETCHED(状态意思为没有采集过))


bst g22 jinniu lilai opebet orange88 vinbet xbet yuebo zunlong shijiebei bet007 hg0088 ju111 letiantang m88 mayaba qg777 qianyiguoji sbf777 tengbohui tlc ule weilianxier waiweitouzhu xingfayule xinhaotiandi yinheyule youfayule zhongying 2018shijiebei w88 18luck 188bet beplay manbet 12bet 95zz shenbo weide1946 ca88 88bifa aomenxinpujing betway bodog bt365 bwin tongbao vwin weinisiren 88jt fenghuangyule hongyunguoji 918botiantang huanyayule jianada28 jixiangfang libo long8 hongzuyishi zuqiutouzhu