2010-2016精選論文2014-d14-1162

1、Proceedingsofthe2014ConferenceonEmpiricalMethodsinNaturalLanguageProcessing(EMNLP)pages1532–1543October25292014DohaQatar.c?2014AssociationfComputationalLinguisticsGloVe:GlobalVectsfWdRepresentationJeffreyPenningtonRidSoc

2、herChristopherD.ManningComputerScienceDepartmentStanfdUniversityStanfdCA94305bstractRecentmethodsflearningvectspacerepresentationsofwdshavesucceededincapturingfinegrainedsemanticsyntacticregularitiesusingvectarithmeticbu

3、ttheiginoftheseregularitieshasremainedopaque.Weanalyzemakeexplicitthemodelpropertiesneededfsuchregularitiestoemergeinwdvects.Theresultisanewgloballogbilinearregressionmodelthatcombinestheadvantagesofthetwomajmodelfamilie

4、sintheliterature:globalmatrixfactizationlocalcontextwindowmethods.Ourmodelefficientlyleveragesstatisticalinfmationbytrainingonlyonthenonzeroelementsinawdwdcooccurrencematrixratherthanontheentiresparsematrixonindividualco

5、ntextwindowsinalargecpus.Themodelproducesavectspacewithmeaningfulsubstructureasevidencedbyitsperfmanceof75%onarecentwdanalogytask.Italsooutperfmsrelatedmodelsonsimilaritytasksnamedentityrecognition.1IntroductionSemanticv

6、ectspacemodelsoflanguagerepresenteachwdwitharealvaluedvect.Thesevectscanbeusedasfeaturesinavarietyofapplicationssuchasinfmationretrieval(Manningetal.2008)documentclassification(Sebastiani2002)questionanswering(Tellexetal

7、.2003)namedentityrecognition(Turianetal.2010)parsing(Socheretal.2013).Mostwdvectmethodsrelyonthedistanceanglebetweenpairsofwdvectsastheprimarymethodfevaluatingtheintrinsicqualityofsuchasetofwdrepresentations.RecentlyMiko

8、lovetal.(2013c)introducedanewevaluationschemebasedonwdanalogiesthatprobesthefinerstructureofthewdvectspacebyexaminingnotthescalardistancebetweenwdvectsbutrathertheirvariousdimensionsofdifference.Fexampletheanalogy“kingis

9、toqueenasmanistowoman”shouldbeencodedinthevectspacebythevectequationking?queen=man?woman.Thisevaluationschemefavsmodelsthatproducedimensionsofmeaningtherebycapturingthemulticlusteringideaofdistributedrepresentations(Beng

10、io2009).Thetwomainmodelfamiliesflearningwdvectsare:1)globalmatrixfactizationmethodssuchaslatentsemanticanalysis(LSA)(Deerwesteretal.1990)2)localcontextwindowmethodssuchastheskipgrammodelofMikolovetal.(2013c).Currentlybot

11、hfamiliessuffersignificantdrawbacks.WhilemethodslikeLSAefficientlyleveragestatisticalinfmationtheydorelativelypolyonthewdanalogytaskindicatingasuboptimalvectspacestructure.Methodslikeskipgrammaydobetterontheanalogytaskbu

12、ttheypolyutilizethestatisticsofthecpussincetheytrainonseparatelocalcontextwindowsinsteadofonglobalcooccurrencecounts.Inthiswkweanalyzethemodelpropertiesnecessarytoproducelineardirectionsofmeaningarguethatgloballogbilinea

13、rregressionmodelsareappropriatefdoingso.Weproposeaspecificweightedleastsquaresmodelthattrainsonglobalwdwdcooccurrencecountsthusmakesefficientuseofstatistics.Themodelproducesawdvectspacewithmeaningfulsubstructureasevidenc

14、edbyitsstateoftheartperfmanceof75%accuracyonthewdanalogydataset.Wealsodemonstratethatourmethodsoutperfmothercurrentmethodsonseveralwdsimilaritytasksalsoonacommonnamedentityrecognition(NER)benchmark.Weprovidethesourcecode

15、fthemodelaswellastrainedwdvectsat:nlp.stanfd.eduprojectsglove.1532Table1:Cooccurrenceprobabilitiesftargetwdsicesteamwithedcontextwdsfroma6billiontokencpus.Onlyintheratiodoesnoisefromnondiscriminativewdslikewaterfashionca

16、nceloutsothatlargevalues(muchgreaterthan1)crelatewellwithpropertiesspecifictoicesmallvalues(muchlessthan1)crelatewellwithpropertiesspecificofsteam.ProbabilityRatiok=solidk=gask=waterk=fashionP(k|ice)1.910?46.610?53.010?3

17、1.710?5P(k|steam)2.210?57.810?42.210?31.810?5P(k|ice)P(k|steam)8.98.510?21.360.96contextofwdi.Webeginwithasimpleexamplethatshowcaseshowcertainaspectsofmeaningcanbeextracteddirectlyfromcooccurrenceprobabilities.Considertw

18、owdsijthatexhibitaparticularaspectofinterestfconcretenesssupposeweareinterestedintheconceptofthermodynamicphasefwhichwemighttakei=icej=steam.Therelationshipofthesewdscanbeexaminedbystudyingtheratiooftheircooccurrenceprob

19、abilitieswithvariousprobewdsk.Fwdskrelatedtoicebutnotsteamsayk=solidweexpecttheratioPikPjkwillbelarge.Similarlyfwdskrelatedtosteambutnoticesayk=gastheratioshouldbesmall.Fwdsklikewaterfashionthatareeitherrelatedtobothices

20、teamtoneithertheratioshouldbeclosetoone.Table1showstheseprobabilitiestheirratiosfalargecpusthenumbersconfirmtheseexpectations.Comparedtotherawprobabilitiestheratioisbetterabletodistinguishrelevantwds(solidgas)fromirrelev

21、antwds(waterfashion)itisalsobetterabletodiscriminatebetweenthetworelevantwds.Theaboveargumentsuggeststhattheappropriatestartingpointfwdvectlearningshouldbewithratiosofcooccurrenceprobabilitiesratherthantheprobabilitiesth

22、emselves.NotingthattheratioPikPjkdependsonthreewdsijkthemostgeneralmodeltakesthefmF(wiwj?wk)=PikPjk(1)wherew∈Rdarewdvects?w∈RdareseparatecontextwdvectswhoserolewillbediscussedinSection4.2.Inthisequationtherighthsideisext

23、ractedfromthecpusFmaydependonsomeasofyetunspecifiedparameters.ThenumberofpossibilitiesfFisvastbutbyenfcingafewdesideratawecanauniquechoice.FirstwewouldlikeFtoencodetheinfmationpresenttheratioPikPjkinthewdvectspace.Sincev

24、ectspacesareinherentlylinearstructuresthemostnaturalwaytodothisiswithvectdifferences.WiththisaimwecanrestrictourconsiderationtothosefunctionsFthatdependonlyonthedifferenceofthetwotargetwdsmodifyingEqn.(1)toF(wi?wj?wk)=Pi

25、kPjk.(2)NextwenotethattheargumentsofFinEqn.(2)arevectswhiletherighthsideisascalar.WhileFcouldbetakentobeacomplicatedfunctionparameterizedbye.g.aneuralwkdoingsowouldobfuscatethelinearstructurewearetryingtocapture.Toavoidt

26、hisissuewecanfirsttakethedotproductoftheargumentsF?(wi?wj)T?wk?=PikPjk(3)whichpreventsFfrommixingthevectdimensionsinundesirableways.Nextnotethatfwdwdcooccurrencematricesthedistinctionbetweenawdacontextwdisarbitrarythatwe

27、arefreetoexchangethetworoles.Todosoconsistentlywemustnotonlyexchangew??wbutalsoX?XT.OurfinalmodelshouldbeinvariantunderthisrelabelingbutEqn.(3)isnot.Howeverthesymmetrycanberestedintwosteps.FirstwerequirethatFbeahomomphis