the state of physical attacks on deep learning systems · lujo bauer mila jovovich (87%) sharif et...

TheStateofPhysicalAttacksonDeepLearningSystems

EarlenceFernandes

Collaborators:IvanEvtimov,KevinEykholt,Chaowei Xiao,AmirRahmati,FlorianTramer,BoLi,AtulPrakash,Tadayoshi Kohno,DawnSong

ImagerecognitionObjectdetection

ScenesegmentationDNAvariantcalling

GameplayingSpeechrecognition

Re-enactingpoliticiansColorizingphotosPoseestimationDescribingphotosGeneratingphotos

TranslationMusiccompositions

CreatingartCreatingDNNs

PredictingearthquakesParticlephysics

QuantumchemistryRecommendationsCreatingfakenewsFightingfakenews

NLPAutomatedSurveillance

…

DeepLearning+Cyber-PhysicalSystems

AirborneCollisionAvoidanceSystemXunmanned(ACASXu)

Apollo(Baidu)Self-DrivingCar

TheGibbon-ImpersonatingPandaaka,AdversarialExamples

“panda”57.7%confidence

“gibbon”99.3%confidence

ImageCredit:OpenAI

=+ ε

ExplainingandHarnessingAdversarialExamples,Goodfellow etal.,arXiv 1412.6572,2015

But,anattackerrequirespixel-leveldigitalaccesstothemodel’sinput

Howcanattackerscreatephysicalattacks?

ACompendiumofPhysicalAttacksPrintingoutadigitallycreatedadversarialexampleworks,butislessrobusttoenvironmentalconditions

Printedpatternsoneyeglass-shapedcut-outscancompromisefacerecognition

clean adversarial

Kurakin etal.,AdversarialExamplesinthePhysicalWorld,arXiv 1607.02533,2016

FastGradientSignMethod(FGSM)approach

Lujo Bauer MilaJovovich(87%)

Sharifetal.,AccessorizetoaCrime:RealandStealthyAttacksonState-of-the-ArtFaceRecognition,CCS2016

Optimizationapproach

ACompendiumofPhysicalAttacksStickersonStopsignscanfoolobjectclassifiersand

detectorsinarangeofphysicalconditions

Optimizationapproach

Eykholt etal.,RobustPhysical-WorldAttacksonDeepLearningVisualClassification,CVPR2018

Eykholt etal.,PhysicalAdversarialExamplesforObjectDetectors,WOOT2018

Mywork

Chenetal.,RobustPhysicalAdversarialAttackonFaster-RCNNObjectDetector,arXiv 1804.05810,2018

AttackerscanbackdoorDNNssothatspecialstickerscausespecificbehavior

Training-timeattack

Guetal.,BadNets:IdentifyingVulnerabilitiesintheMachineLearningModelSupplyChain,arXiv1708.06733,2017

ACompendiumofPhysicalAttacks3Dprintedturtlescanberiflestoastate-of-the-artclassifier

AdversarialExamplescanhideinmusic

Patchesthatcamouflageanyobjectasatoasterexist

Athalye etal.,SynthesizingRobustAdversarialExamples,ICML2018

Expectation-over-Transformationsapproach(optimization)

Expectation-over-Transformationsapproach(optimization)

Brownetal.,AdversarialPatch,arXiv 1712.09665,May2018

Carlini etal.,AudioAdversarialExamples:TargetedAttacksonSpeech-

to-Text,DLSWorkshop2018

Yuanetal.,CommanderSong:ASystematicApproachforPractical

AdversarialVoiceRecognition,USENIXSecurity2018

OpenQuestions

• Arethereotherphysicaldomainswherewecanexploreadversarialexamples?• Currentattacksonlylookatasinglemodel.But,amodelisonlyapartofthewholeCPS.Dotheseattackshavesystem-wideeffects?• Isthereanythingspecificaboutphysicaladversarialexamplesthatmakethemeasierormoredifficulttodefendagainst?• Shouldweonlydependon“pureML”techniquesfordefense?• WhataspectsofCPSscanweleveragetodefend(defenseindepth)?

Thankyou!

• Arethereotherphysicaldomainswherewecanexploreadversarialexamples?• Currentattacksonlylookatasinglemodel.But,amodelisonlyapartofthewholeCPS.Dotheseattackshavesystem-wideeffects?• Isthereanythingspecificaboutphysicaladversarialexamplesthatmakethemeasierormoredifficulttodefendagainst?• Shouldweonlydependon“pureML”techniquesfordefense?• WhataspectsofCPSscanweleveragetodefend(defenseindepth)?

EarlenceFernandes,earlence@cs.washington.edu,earlence.com