基于Scrapy框架的新聞實(shí)時(shí)抓取及處理系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn).pdf

1、南開大學(xué)碩士學(xué)位論文基于Scrapy框架的新聞實(shí)時(shí)抓取及處理系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)姓名：林偉堅(jiān)申請學(xué)位級別：碩士專業(yè)：計(jì)算機(jī)軟件與理論指導(dǎo)教師：袁曉潔201205AbstractAbstractWiththerapiddevelopmentofIntemet，theinformationexplosionoftheIntemetbringstheproblemofinformationoverloadThereleasemethodofnew

2、shasbeenshiftfromtraditionalmediatotheIntemetAsanimportantpartoftheInternetinformation，newsalsoincreasesdramaticallyInordertomakeitmoreeasytogetthenews，thepaperdesignsarealtimedistributednewscrawlertoaggregatethenewsinth

3、eIntemetandapipelinetoprocessthenewsdataThepapersanalyzesandsummarizesthecharacteristicsofIntemetnews，newssitesandnewscrawleranddesignscorealgorithmsofthenewscrawlerbasedonthesecharacteristics，whicharecrawlingstrategyand

4、updatingstrategyBasedontwoopensourcedatabases，RedisandMongoDB，andScrapyCrawlerFramework，thepaperdesignsareal—timedistributednewscrawler，whichusesXPathandregularexpressionstoextractandidentifytherelevantdataThepaperalsoim

5、plementsanewsdataprocessingpipeline，includingnewsdataextraction，propertiesnormalization，newspagescombinerandnewscontentcleaningAccordingtothecharacteristicsofthenews，thepaperusesShinglesasthefeatureextractionmethodandGoo

6、gle’SSimHashasthenearduplicatesdetectingalgorithmThenewscrawlerandprocessingpipelinehavebeensetupinactualproductionBycrawlingandmonitoringmorethan120newssitesandmorethan1300initialnavigationpage，everydaythesystemisableto