基于lucene面向主题爬行搜索引擎应用地研究.pdf

摘要摘要基于主题爬行的搜索引擎,即垂直搜索引擎,又可称为主题搜索引擎。它与通用搜索引擎的区别主要体现在资源与服务上,主题搜索引擎提供的资源都是与某个主题相关的,服务上更具有专业特色,有一定程度的个性化。由于只针对某一特定主题进行深入挖掘,所以在特定主题上分类更加细致更加精确,数据也更加全面和深入,资源的更新也相对更加及时。Web资源日新月异的发展,更新和多元化发展的速度让搜索引擎面临着一个尴尬的境地。通用搜索引擎己无法满足特定用户更深入的查询需求,因而针对特定领域进行主题爬行的垂直搜索引擎将会越来越实用以及可行。本文主要对垂直搜索引擎的核心主题爬行进行研究,它的关键技术包括网页分类技术、待爬行URLs优先级计算以及主题爬行算法。本文针对关键技术介绍了基于数据挖掘的网页分类方法,并提出了基于网页分块的待爬行URLs优先级计算方法,通过利用网页分块技术,以块为单位来计算待爬行URLs的主题相关性,通过整个块的文本来预测待爬行URLs与主题的相关度,这样就可以解决单个链接的锚文本包含的信息太少,对网页与主题的相关度判断有所限制。并且可以更好的过滤噪音块和与主题不相关的链接块,在一定程度上提高主题爬行效率。最后设计并构建出了一个面向电脑产品信息的垂直搜索引擎系统,主题爬行模块主要是构建了基于SVM的二元分类器,判断当前网页的主题相关性,提高了主题判断的精确性。基于Lucene检索框架设计并实现了信息处理模块和信息检索模块,为整个搜索引擎的设计提供了很大的方便。图17表2参45 关键词:主题爬行;搜索引擎;Lucene;网页分块;数据挖掘分类号:TP39 摘要 Abstract Verticalsearch engine is atype ofsearchengine(SE)based on focused crawling, which isalsocalledtopic—specific differsfrom general-purpose SE mainly in resources and services itprovides,which arerelated tocertaintopic,thus it’Smore professional and personalized tosome —specific search engine search the web resources on thetopic—specific field,SO theclassification on theparticulartopic is more accurate prehensive,the resources arefullandin-depth,the update ofthe resources ismore rapiddevelopment oftheWeb made search engines face a —purpose search engines have been unable tosatisfy the user’S thetopic—specific search engines willbemore practicaland feasible. The paper mainly studied on thefocused crawling,which isthe core ofvertical search key technologies of focused crawling include web page classification puting thepriorities oftheURLs which tobecrawled and thealgorithm offocused is tobeintroduced themethod ofWeb page classification based on data mining,and also put up inpaper the method puting thepriorities oftheTCURLs(URLs to becrawling)based on page thetechnology pute thetopic correlation ofTCURLs inblocks the whole the context ofblocks pute the TCURLs priority,w