畢業(yè)論文外文翻譯-網(wǎng)絡(luò)爬蟲(chóng)

1、Crawlingthewebisdeceptivelysimple:thebasicalgithmis(a)Fetchapage(b)ParseittoextractalllinkedURLs(c)FalltheURLsnotseenbeferepeat(a)–(c).Howeverthesizeoftheweb(estimatedatover4billionpages)itsrateofchange(estimatedat7%perw

2、eek)movethisplanfromatrivialprogrammingexercisetoaseriousalgithmicsystemdesignchallenge.Indeedthesetwofactsaloneimplythatfareasonablyfreshcompletecrawlofthewebstep(a)mustbeexecutedaboutathoustimespersecondthusthemembersh

3、iptest(c)mustbedonewellovertenthoustimespersecondagainstasettoolargetosteinmainmemy.Thisrequiresadistributedarchitecturewhichfurthercomplicatesthemembershiptest.Acrucialwaytospeedupthetestistocachethatistosteinmainmemya(

4、dynamic)subsetofthe“seen”URLs.ThemaingoalofthispaperistocarefullyinvestigateseveralURLcachingtechniquesfwebcrawling.Weconsiderbothpracticalalgithms:romreplacementstaticcacheLRUCLOCKtheeticallimits:clairvoyantcachinginfin

5、itecache.Weperfmedabout1800simulationsusingthesealgithmswithvariouscachesizesusingactuallogdataextractedfromamassive33daywebcrawlthatissuedoveronebillionHTTPrequests.Ourmainconclusionisthatcachingisveryeffective–inourset

6、upacacheofroughly50000entriescanachieveahitrateofalmost80%.Interestinglythiscachesizefallsatacriticalpoint:asubstantiallysmallercacheismuchlesseffectivewhileasubstantiallylargercachebringslittleadditionalbenefit.Weconjec

7、turethatsuchcriticalpointsareinherenttoourproblemventureanexplanationfthisphenomenon.1.INTRODUCTIONArecentPewFoundationstudy[31]statesthat“SearchengineshavebecomeanindispensableutilityfInters”estimatesthatasof2002slightl

8、yover50%ofallAmericanshaveusedwebsearchtofindinfmation.Hencethetechnologythatpowerswebsearchisofenmouspracticalinterest.Inthispaperweconcentrateononeaspectofthesearchtechnologynamelytheprocessofcollectingsnapshotoftheweb

9、asearchenginemustcrawlatleast100millionpagesperday.Therefestep(a)mustbeexecutedabout1000timespersecondthemembershiptestinstep(c)mustbedonewellovertenthoustimespersecondagainstasetofURLsthatistoolargetosteinmainmemy.Inadd

10、itioncrawlerstypicallyuseadistributedarchitecturetocrawlmepagesinparallelwhichfurthercomplicatesthemembershiptest:itispossiblethatthemembershipquestioncanonlybeansweredbyapeernodenotlocally.Acrucialwaytospeedupthemembers

11、hiptestistocachea(dynamic)subsetofthe“seen”URLsinmainmemy.ThemaingoalofthispaperistoinvestigateindepthseveralURLcachingtechniquesfwebcrawling.Weexaminedfourpracticaltechniques:romreplacementstaticcacheLRUCLOCKcomparedthe

12、magainsttwotheeticallimits:clairvoyantcachinginfinitecachewhenrunagainstatraceofawebcrawlthatissuedoveronebillionHTTPrequests.Wefoundthatsimplecachingtechniquesareextremelyeffectiveevenatrelativelysmallcachesizessuchas50

13、000entriesshowhowthesecachescanbeimplementedveryefficiently.Thepaperisganizedasfollows:Section2discussesthevariouscrawlingsolutionsproposedintheliteraturehowcachingfitsintheirmodel.Section3presentsanintroductiontocaching

14、techniquesdescribesseveraltheeticalpracticalalgithmsfcaching.WeimplementedthesealgithmsundertheexperimentalsetupdescribedinSection4.TheresultsofoursimulationsaredepicteddiscussedinSection5ourrecommendationsfpracticalalgi

15、thmsdatastructuresfURLcachingarepresentedinSection6.Section7containsourconclusionsdirectionsffurtherresearch.2.CRAWLINGWebcrawlersarealmostasoldasthewebitselfnumerouscrawlingsystemshavebeendescribedintheliterature.Inthis