robust estimation in parameter learning

RobustEstimationinParameterLearning

SimonsInstituteBootcamp Tutorial,Part2

AnkurMoitra(MIT)

CLASSICPARAMETERLEARNINGGivensamplesfromanunknowndistributioninsomeclass

e.g.a1-DGaussian

canweaccuratelyestimateitsparameters?

CLASSICPARAMETERLEARNINGGivensamplesfromanunknowndistributioninsomeclass

e.g.a1-DGaussian

canweaccuratelyestimateitsparameters? Yes!

CLASSICPARAMETERLEARNINGGivensamplesfromanunknowndistributioninsomeclass

e.g.a1-DGaussian

canweaccuratelyestimateitsparameters?

empiricalmean: empiricalvariance:

Yes!

Themaximumlikelihoodestimatorisasymptoticallyefficient(1910-1920)

R.A.Fisher

Themaximumlikelihoodestimatorisasymptoticallyefficient(1910-1920)

R.A.Fisher J.W.Tukey

Whatabouterrors inthemodelitself?(1960)

ROBUSTPARAMETERLEARNINGGivencorrupted samplesfroma1-DGaussian:

canweaccuratelyestimateitsparameters?

=+idealmodel noise observedmodel

Howdoweconstrainthenoise?

Howdoweconstrainthenoise?

Equivalently:

L1-normofnoiseatmostO(ε)

Howdoweconstrainthenoise?

Equivalently:

L1-normofnoiseatmostO(ε) ArbitrarilycorruptO(ε)-fractionofsamples(inexpectation)

Howdoweconstrainthenoise?

Equivalently:

ThisgeneralizesHuber’sContaminationModel:Anadversarycanadd anε-fractionofsamples

L1-normofnoiseatmostO(ε) ArbitrarilycorruptO(ε)-fractionofsamples(inexpectation)

Howdoweconstrainthenoise?

Equivalently:

ThisgeneralizesHuber’sContaminationModel:Anadversarycanadd anε-fractionofsamples

L1-normofnoiseatmostO(ε) ArbitrarilycorruptO(ε)-fractionofsamples(inexpectation)

Outliers:Pointsadversaryhascorrupted,Inliers:Pointshehasn’t

Inwhatnormdowewanttheparameterstobeclose?

Inwhatnormdowewanttheparameterstobeclose?

Definition:Thetotalvariationdistancebetweentwodistributionswithpdfs f(x)andg(x)is

Inwhatnormdowewanttheparameterstobeclose?

FromtheboundontheL1-normofthenoise,wehave:

observedideal

Definition:Thetotalvariationdistancebetweentwodistributionswithpdfs f(x)andg(x)is

Inwhatnormdowewanttheparameterstobeclose?

Definition:Thetotalvariationdistancebetweentwodistributionswithpdfs f(x)andg(x)is

estimate ideal

Goal:Finda1-DGaussianthatsatisfies

Inwhatnormdowewanttheparameterstobeclose?

estimate observed

Definition:Thetotalvariationdistancebetweentwodistributionswithpdfs f(x)andg(x)is

Equivalently,finda1-DGaussianthatsatisfies

Dotheempiricalmeanandempiricalvariancework?

Dotheempiricalmeanandempiricalvariancework?

No!

Dotheempiricalmeanandempiricalvariancework?

No!

=+idealmodel noise observedmodel

Dotheempiricalmeanandempiricalvariancework?

No!

=+idealmodel noise observedmodel

Butthemedian andmedianabsolutedeviationdowork

Fact[Folklore]:Givensamplesfromadistributionthatisε-closeintotalvariationdistancetoa1-DGaussian

themedianandMADrecoverestimatesthatsatisfy

where

Fact[Folklore]:Givensamplesfromadistributionthatisε-closeintotalvariationdistancetoa1-DGaussian

themedianandMADrecoverestimatesthatsatisfy

where

Alsocalled(properly)agnosticallylearninga1-DGaussian

Fact[Folklore]:Givensamplesfromadistributionthatisε-closeintotalvariationdistancetoa1-DGaussian

themedianandMADrecoverestimatesthatsatisfy

where

Whataboutrobustestimationinhigh-dimensions?

Whataboutrobustestimationinhigh-dimensions?

e.g.microarrayswith10kgenes

Fact[Folklore]:Givensamplesfromadistributionthatisε-closeintotalvariationdistancetoa1-DGaussian

themedianandMADrecoverestimatesthatsatisfy

where

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

MainProblem:Givensamplesfromadistributionthatisε-closeintotalvariationdistancetoad-dimensionalGaussian

giveanefficientalgorithmtofindparametersthatsatisfy

MainProblem:Givensamplesfromadistributionthatisε-closeintotalvariationdistancetoad-dimensionalGaussian

giveanefficientalgorithmtofindparametersthatsatisfy

SpecialCases:

(1)Unknownmean

(2)Unknowncovariance

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

UnknownMean

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

O(ε)

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

O(ε) NP-Hard

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

O(ε) NP-Hard

GeometricMedian

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

O(ε) NP-Hard

GeometricMedian poly(d,N)

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

O(ε) NP-Hard

GeometricMedian poly(d,N)O(ε√d)

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

O(ε) NP-Hard

GeometricMedian poly(d,N)O(ε√d)

Tournament O(ε) NO(d)

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian

UnknownMean

O(ε) NP-Hard

GeometricMedian poly(d,N)O(ε√d)

Tournament O(ε) NO(d)

O(ε√d)Pruning O(dN)

ACOMPENDIUMOFAPPROACHES

ErrorGuarantee

RunningTime

TukeyMedian O(ε) NP-Hard

GeometricMedian O(ε√d) poly(d,N)

Tournament O(ε) NO(d)

O(ε√d)Pruning O(dN)

UnknownMean

…

ThePriceofRobustness?

Allknownestimatorsarehardtocomputeorlosepolynomial factorsinthedimension

ThePriceofRobustness?

Allknownestimatorsarehardtocomputeorlosepolynomial factorsinthedimension

Equivalently:Computationallyefficientestimatorscanonlyhandle

fractionoferrorsandgetnon-trivial(TV<1)guarantees

ThePriceofRobustness?

Allknownestimatorsarehardtocomputeorlosepolynomial factorsinthedimension

Equivalently:Computationallyefficientestimatorscanonlyhandle

fractionoferrorsandgetnon-trivial(TV<1)guarantees

ThePriceofRobustness?

Allknownestimatorsarehardtocomputeorlosepolynomial factorsinthedimension

Equivalently:Computationallyefficientestimatorscanonlyhandle

fractionoferrorsandgetnon-trivial(TV<1)guarantees

Isrobustestimationalgorithmicallypossibleinhigh-dimensions?

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

RECENTRESULTS

Theorem[Diakonikolas,Li,Kamath,Kane,Moitra,Stewart‘16]:Thereisanalgorithmwhengivensamplesfromadistributionthatisε-closeintotalvariationdistancetoad-dimensionalGaussianfindsparametersthatsatisfy

Robustestimationishigh-dimensionsisalgorithmicallypossible!

Moreoverthealgorithmrunsintimepoly(N,d)

RECENTRESULTS

Theorem[Diakonikolas,Li,Kamath,Kane,Moitra,Stewart‘16]:Thereisanalgorithmwhengivensamplesfromadistributionthatisε-closeintotalvariationdistancetoad-dimensionalGaussianfindsparametersthatsatisfy

Robustestimationishigh-dimensionsisalgorithmicallypossible!

Moreoverthealgorithmrunsintimepoly(N,d)

Extensions:Canweakenassumptionstosub-Gaussianorboundedsecondmoments(withweakerguarantees)forthemean

Independentlyandconcurrently:

Theorem[Lai,Rao,Vempala ‘16]:Thereisanalgorithmwhengivensamplesfromadistributionthatisε-closeintotal

variationdistancetoad-dimensionalGaussianfindsparametersthatsatisfy

Moreoverthealgorithmrunsintimepoly(N,d)

Independentlyandconcurrently:

Theorem[Lai,Rao,Vempala ‘16]:Thereisanalgorithmwhengivensamplesfromadistributionthatisε-closeintotal

variationdistancetoad-dimensionalGaussianfindsparametersthatsatisfy

Moreoverthealgorithmrunsintimepoly(N,d)

Whenthecovarianceisbounded,thistranslatesto:

AGENERALRECIPE

Robustestimationinhigh-dimensions:

� Step#1:Findanappropriateparameterdistance

� Step#2:Detectwhenthenaïveestimatorhasbeencompromised

� Step#3:Findgoodparameters,ormakeprogressFiltering:FastandpracticalConvexProgramming:Bettersamplecomplexity

AGENERALRECIPE

Robustestimationinhigh-dimensions:

� Step#1:Findanappropriateparameterdistance

� Step#2:Detectwhenthenaïveestimatorhasbeencompromised

� Step#3:Findgoodparameters,ormakeprogressFiltering:FastandpracticalConvexProgramming:Bettersamplecomplexity

Let’sseehowthisworksforunknownmean…

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

ABasicFact:

(1)

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

ABasicFact:

(1)

ThiscanbeprovenusingPinsker’s Inequality

andthewell-knownformulaforKL-divergencebetweenGaussians

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

ABasicFact:

(1)

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

ABasicFact:

(1)

Corollary:Ifourestimate(intheunknownmeancase)satisfies

then

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

ABasicFact:

(1)

Corollary:Ifourestimate(intheunknownmeancase)satisfies

then

OurnewgoalistobecloseinEuclideandistance

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

DETECTINGCORRUPTIONS

Step#2:Detectwhenthenaïveestimatorhasbeencompromised

DETECTINGCORRUPTIONS

Step#2:Detectwhenthenaïveestimatorhasbeencompromised

=uncorrupted=corrupted

DETECTINGCORRUPTIONS

Step#2:Detectwhenthenaïveestimatorhasbeencompromised

=uncorrupted=corrupted

Thereisadirectionoflarge(>1)variance

KeyLemma:IfX1,X2,…XN comefromadistributionthatisε-closetoandthenfor

(1) (2)

withprobabilityatleast1-δ

KeyLemma:IfX1,X2,…XN comefromadistributionthatisε-closetoandthenfor

(1) (2)

withprobabilityatleast1-δ

Take-away:Anadversaryneedstomessupthesecondmomentinordertocorruptthefirstmoment

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints:

v

wherevisthedirectionoflargestvariance

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints:

v

wherevisthedirectionoflargestvariance,andThasaformula

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints:

v

T

wherevisthedirectionoflargestvariance,andThasaformula

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints

Ifwecontinuetoolong,we’dhavenocorruptedpointsleft!

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints

Ifwecontinuetoolong,we’dhavenocorruptedpointsleft!

Eventuallywefind(certifiably)goodparameters

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints

Ifwecontinuetoolong,we’dhavenocorruptedpointsleft!

Eventuallywefind(certifiably)goodparameters

RunningTime: SampleComplexity:

AWIN-WINALGORITHM

Step#3:Eitherfindgoodparameters,orremovemanyoutliers

FilteringApproach:Supposethat:

Wecanthrowoutmorecorruptedthanuncorruptedpoints

Ifwecontinuetoolong,we’dhavenocorruptedpointsleft!

Eventuallywefind(certifiably)goodparameters

RunningTime: SampleComplexity:ConcentrationofLTFs

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

AGENERALRECIPE

Robustestimationinhigh-dimensions:

� Step#1:Findanappropriateparameterdistance

� Step#2:Detectwhenthenaïveestimatorhasbeencompromised

� Step#3:Findgoodparameters,ormakeprogressFiltering:FastandpracticalConvexProgramming:Bettersamplecomplexity

AGENERALRECIPE

Robustestimationinhigh-dimensions:

� Step#1:Findanappropriateparameterdistance

� Step#2:Detectwhenthenaïveestimatorhasbeencompromised

� Step#3:Findgoodparameters,ormakeprogressFiltering:FastandpracticalConvexProgramming:Bettersamplecomplexity

Howaboutforunknowncovariance?

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

AnotherBasicFact:

(2)

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

AnotherBasicFact:

Again,provenusingPinsker’s Inequality

(2)

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

AnotherBasicFact:

Again,provenusingPinsker’s Inequality

(2)

Ournewgoalistofindanestimatethatsatisfies:

PARAMETERDISTANCE

Step#1:FindanappropriateparameterdistanceforGaussians

AnotherBasicFact:

Again,provenusingPinsker’s Inequality

(2)

Ournewgoalistofindanestimatethatsatisfies:

Distanceseemsstrange,butit’stherightonetousetoboundTV

UNKNOWNCOVARIANCE

Whatifwearegivensamplesfrom?

UNKNOWNCOVARIANCE

Whatifwearegivensamplesfrom?

Howdowedetectifthenaïveestimatoriscompromised?

UNKNOWNCOVARIANCE

Whatifwearegivensamplesfrom?

Howdowedetectifthenaïveestimatoriscompromised?

KeyFact:Let and

Thenrestrictedtoflattenings ofdxdsymmetricmatrices

UNKNOWNCOVARIANCE

Whatifwearegivensamplesfrom?

Howdowedetectifthenaïveestimatoriscompromised?

KeyFact:Let and

Thenrestrictedtoflattenings ofdxdsymmetricmatrices

ProofusesIsserlis’s Theorem

UNKNOWNCOVARIANCE

needtoprojectout

Whatifwearegivensamplesfrom?

Howdowedetectifthenaïveestimatoriscompromised?

KeyFact:Let and

Thenrestrictedtoflattenings ofdxdsymmetricmatrices

KeyIdea: Transformthedata,lookforrestrictedlargeeigenvalues

KeyIdea: Transformthedata,lookforrestrictedlargeeigenvalues

KeyIdea: Transformthedata,lookforrestrictedlargeeigenvalues

Ifwerethetruecovariance,wewouldhaveforinliers

KeyIdea: Transformthedata,lookforrestrictedlargeeigenvalues

Ifwerethetruecovariance,wewouldhaveforinliers,inwhichcase:

wouldhavesmallrestrictedeigenvalues

KeyIdea: Transformthedata,lookforrestrictedlargeeigenvalues

Ifwerethetruecovariance,wewouldhaveforinliers,inwhichcase:

wouldhavesmallrestrictedeigenvalues

Take-away:Anadversaryneedstomessupthe(restricted)fourthmomentinordertocorruptthesecondmoment

ASSEMBLINGTHEALGORITHM

Givensamplesthatareε-closeintotalvariationdistancetoad-dimensionalGaussian

ASSEMBLINGTHEALGORITHM

Givensamplesthatareε-closeintotalvariationdistancetoad-dimensionalGaussian

Step#1:Doublingtrick

ASSEMBLINGTHEALGORITHM

Givensamplesthatareε-closeintotalvariationdistancetoad-dimensionalGaussian

Step#1:Doublingtrick

Nowusealgorithmforunknowncovariance

ASSEMBLINGTHEALGORITHM

Givensamplesthatareε-closeintotalvariationdistancetoad-dimensionalGaussian

Step#1:Doublingtrick

Nowusealgorithmforunknowncovariance

Step#2:(Agnostic)isotropicposition

ASSEMBLINGTHEALGORITHM

Givensamplesthatareε-closeintotalvariationdistancetoad-dimensionalGaussian

Step#1:Doublingtrick

Nowusealgorithmforunknowncovariance

Step#2:(Agnostic)isotropicposition

rightdistance,ingeneralcase

ASSEMBLINGTHEALGORITHM

Givensamplesthatareε-closeintotalvariationdistancetoad-dimensionalGaussian

Step#1:Doublingtrick

Nowusealgorithmforunknowncovariance

Step#2:(Agnostic)isotropicposition

Nowusealgorithmforunknownmeanrightdistance,ingeneralcase

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

PartI:Introduction

� RobustEstimationinOne-dimension� Robustnessvs.HardnessinHigh-dimensions

� RecentResults

PartII:AgnosticallyLearningaGaussian

� ParameterDistance� DetectingWhenanEstimatorisCompromised

� AWin-WinAlgorithm� UnknownCovariance

OUTLINE

PartIII:FurtherResults

LIMITSTOROBUSTESTIMATION

Theorem[Diakonikolas,Kane,Stewart‘16]:Anystatisticalquerylearning* algorithminthestrongcorruptionmodel

thatmakeserrormustmakeatleastqueries

insertionsanddeletions

LIMITSTOROBUSTESTIMATION

Theorem[Diakonikolas,Kane,Stewart‘16]:Anystatisticalquerylearning* algorithminthestrongcorruptionmodel

thatmakeserrormustmakeatleastqueries

*Insteadofseeingsamplesdirectly,analgorithmqueriesafnctn

andgetsexpectation,uptosamplingnoise

insertionsanddeletions

LISTDECODING

Whatifanadversarycancorruptthemajority ofsamples?

LISTDECODING

Whatifanadversarycancorruptthemajority ofsamples?

Thisextendstomixturesstraightforwardly

Theorem[Charikar,Steinhardt,Valiant‘17]:Givensamplesfromadistributionwithmeanandcovariancewherehavebeencorrupted,thereisanalgorithmthatoutputs

with thatsatisfies

LISTDECODING

Whatifanadversarycancorruptthemajority ofsamples?

Thisextendstomixturesstraightforwardly

Theorem[Charikar,Steinhardt,Valiant‘17]:Givensamplesfromadistributionwithmeanandcovariancewherehavebeencorrupted,thereisanalgorithmthatoutputs

with thatsatisfies

[Kothari,Steinhardt‘18],[Diakonikolas etal’18] gaveimprovedguarantees,butunderGaussianity

BEYONDGAUSSIANS

Canwerelaxthedistributionalassumptions?

BEYONDGAUSSIANS

Theorem[Kothari,Steurer ‘18] [Hopkins,Li’18]:Givenε-corruptedsamplesfromak-certifiablysubgaussian distributionthereisanalgorithmthatoutputs

Canwerelaxthedistributionalassumptions?

BEYONDGAUSSIANS

Theorem[Kothari,Steurer ‘18] [Hopkins,Li’18]:Givenε-corruptedsamplesfromak-certifiablysubgaussian distributionthereisanalgorithmthatoutputs

Canwerelaxthedistributionalassumptions?

Whenyouonlyknowboundsonthemoments,theseguaranteesareoptimal

SUBGAUSSIAN CONFIDENCEINTERVALS

Estimatingthemeanaccuratelywithheavytaileddistributions?

SUBGAUSSIAN CONFIDENCEINTERVALS

Estimatingthemeanaccuratelywithheavytaileddistributions?

Theorem[Hopkins‘18]:Givenniid samplesfromadistributionwithmeanandcovarianceandtargetconfidence,thereisapolynomialtimealgorithmthatoutputssatisfying

SUBGAUSSIAN CONFIDENCEINTERVALS

Estimatingthemeanaccuratelywithheavytaileddistributions?

Theorem[Hopkins‘18]:Givenniid samplesfromadistributionwithmeanandcovarianceandtargetconfidence,thereisapolynomialtimealgorithmthatoutputssatisfying

Theempiricalmeandoesn’twork,andmedian-of-meansestimatordueto [Lugosi,Mendelson ‘18]ishardtocompute

SUBGAUSSIAN CONFIDENCEINTERVALS

Estimatingthemeanaccuratelywithheavytaileddistributions?

Theorem[Hopkins‘18]:Givenniid samplesfromadistributionwithmeanandcovarianceandtargetconfidence,thereisapolynomialtimealgorithmthatoutputssatisfying

Theempiricalmeandoesn’twork,andmedian-of-meansestimatordueto [Lugosi,Mendelson ‘18]ishardtocompute

[Cherapanamjeri,Flammarion,Bartlett‘19]gavefasteralgorithmsbasedongradientdescent

Summary:� Nearlyoptimalalgorithmforagnosticallylearningahigh-dimensionalGaussian

� Generalrecipeusingrestrictedeigenvalueproblems� Furtherapplicationstoothermixturemodels�What’snextforalgorithmicrobuststatistics?

Thanks!AnyQuestions?

Summary:� Nearlyoptimalalgorithmforagnosticallylearningahigh-dimensionalGaussian

� Generalrecipeusingrestrictedeigenvalueproblems� Furtherapplicationstoothermixturemodels�What’snextforalgorithmicrobuststatistics?