aprimerineconometrictheory2016ch14

1、Chapter14RegularizationInseveralsectionsofthisbookwetouchedonthetopicofregularization(seee.g.8.2.1.28.2.3).Avarietyofstatisticalproceduresmachinelearningalgorithmsemployregularization(underdifferentnames)toimproveoutofsa

2、mplefit.Goodoutofsamplefitmeansgeneralizationfromobserveddatawhichaswe’vestressedbefeisthekeyproblemofstatistics.Thischapterintroducesanumberofmethodsthatuseregularizationdiscussestheirstatisticalproperties.14.1Nonparame

3、tricDensityEstimationNonparametricdensityestimationisanapplicationofregularizationtotheproblemofrecoveringdistributionsfromdata.Itcombinesthedatawithapribeliefthatprobabilitymassmostlikelyfallsinplacesotherthanjustthesam

4、plepointsobservedsofar.Aswellasbeingofinterestfromatheeticalperspectivenonparametricdensityestimationisusedinagreatvarietyofappliedstudies.Webeginouranalysiswithareviewofparametricdensityestimationthenproceedtononparamet

5、ricmethods.14.1.1IntroductionSupposeourdataconsistofIIDobservationsx1...xNfromunknowndistributionPonRd.WeassumethroughoutthissectionthatPisabsolutelycontinuous.OuraimistoestimatethedensityofPdenotedbelowbyf.Weknowhowtodo

6、thisinaparametricsetting.Fexamplelet’saddtheassumptionthatfbelongstotheclassofnmaldensitiesonRsothatf=f(σ)=thenmaldensityfdistributionN(σ2).TheMLEsoftheparametersare?N:=377Regularization379(ii)?f?g?1=2supB∈B(Rd)???Bf??Bg

7、??.Theboundin(i)iscalledPinsker’sinequalitywhile(ii)iscalledScheff’sidentity.Scheff’sidentitytellsusthatL1distancemeasuressomethingthatwedirectlycareabout:whenL1deviationissmallsoisthemaximaldeviationbetweenprobabilities

8、assignedtoevents.1Inwhatfollowswewillsaythatasequence?fNofromdensitiesonRdisLpconsistentfadensityfonRdif??fN?f?pp→0asN→∞Example14.1.1Let?fN=f(xNsN)betheNthelementofthesequenceofnmaldensitiesdescribedabovewherex1...xNarei

9、ndependentdrawsfromanmaldensityf=f(σ)xNsNthesamplemeanstarddeviationrespectively.ThissequenceofdensitiesisL1consistentff.Seeexercise14.4.3.14.1.1.1FailureofConsistencyTheriskwiththeparametricapproachisthattheparametricas

10、sumptionisincrectinthesensethattheparametricclassdoesn’tcontainthedensitygeneratingthedataanygoodapproximation.Ifthisisthecaseaparametricapproachistypicallynotconsistent.Mepreciselyifweestimatefwithparametricclassfθθ∈Θth

11、entheLpdeviationbetweenourestimatefisboundedbelowbyδ(f):=infθ∈Θ?f?fθ?p(14.4)Thisvaluewillbezeroonlywhenfcanbeattainedasthelimitofelementsoffθθ∈Θ.Example14.1.2Consideragainthesettingof14.1.1butnowsupposethatthetruedensity

12、fisnotGaussian.Theneitherthesequence?fNisnotL1consistentfanydensityifitisL1consistentfsomedensitythenthatdensityisnotf.Thereasonisthatδ(f)in(14.4)isalwayspositivewhentheparametricclassisGaussianfisnotsincethesetofnmalden

13、sitiesisclosedunderthetakingoflimitsinL1.14.1.2KernelDensityEstimationSometimeswecanmakegoodchoicesfparametricclassesbyusingdeivestatisticsbyappealingtosometheywithsharpquantitativeimplications.Atothertimesthisisdifficul