Upload File

perception deception: physical adversarial attack challenges … · perception deception: physical...

  • Home
  • Documents
  • Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)
38
Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu (Edward) Zhong, Yunhan Jia, Weilin Xu, Tao Wei

Upload: others

Post on 23-May-2020

24 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection

Zhenyu (Edward) Zhong, Yunhan Jia, Weilin Xu, Tao Wei

Page 2: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Our Team X-Lab

Chief Security ScientistDr. Tao Wei

Dr. Yunhan Jia

Dr. Zhenyu(Edward) Zhong edwardzhong [at] baidu DOT com

Weilin Xu

AI Security Research

System Security Research

https://github.com/baidu/rust-sgx-sdk

https://github.com/mesalock-linux

https://github.com/mesalock-linux/mesapy

https://github.com/mesalock-linux/mesalink

https://www.mesatee.org

https://github.com/baidu/AdvBox

Scan Me

Page 3: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Disclaimer

• This talk doesn’t target any commercial autonomous driving systems.

• We don’t provide any comments to the vulnerabilities of the perceptions of existing autonomous driving systems.

• We focus on state-of-the-art object detection methods, all the results/techniques are proof-of-concept.

Page 4: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Car Safety – Unintended Acceleration

https://www.cbsnews.com/news/toyota-unintended-acceleration-has-killed-89/

…the large throttle opening UAs as described in submitted VOQs could not be found…

…does not mean it could not occur…

https://www.eetimes.com/document.asp?doc_id=1319903&page_number=2

…experts identified ”unprotected critical variables.”…

…the defects we found were linked to unintended Acceleration through vehicle testing, …

Page 5: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Car Safety – Autonomous Driving

Page 6: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Car Safety – Rear Ended Into Fire Truck

Page 7: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Imgsrc: http://www.crosslinksolutions.co.uk/service-listing/adas-calibration-equipment/

Car Perception While Driving

$$ $$$$ $Cost Estimate: $$ $

Page 8: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Behind Perception: End2End Object Detection

Object Detection: a technology related to computer vision and image processing that deals with instances of semantic objects of certain class in digital images and videos.

https://software.intel.com/en-us/articles/a-closer-look-at-object-detection-recognition-and-tracking

Page 9: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

State-of-the-Art Vision-based Object Detection

Conv Feature Map YOLO (You Only Look Once)

416

4163

3

3208

208 16

33

104

104 32

3

3

13

13256

3

3

13

13

numAnchors x (5 + numClasses)

3

3

Page 10: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Pick Our Target – YOLOv3

Accuracy on MS COCO

YOLOv3: An Incremental Improvement. Joseph Redmon, Ali Farhadihttps://arxiv.org/pdf/1804.02767.pdf

Page 11: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Pick Our Target – YOLOv3

Performance

YOLOv3: An Incremental Improvement. Joseph Redmon, Ali Farhadihttps://arxiv.org/pdf/1804.02767.pdf

Page 12: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Current Status of Adversarial Example

Definition: For an input image !, minimize " !, ! + % , &. (. ) ! + % = (, ! + % ∈ [0,1]0

The most well-studied distance metric: 12 3456 Perturbations• 78 -- each pixel is allowed to be changed by up to a limit• 79 -- number of pixels altered that matter most• 7: -- many small changes to many pixels

Page 13: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Adversarial Examples & !" NormPerturbations Impact to DNN

#$%& !" based Perturbation Method

Source Image Perturbed Images

'( = ' − + , -./0(2!3--#,5('))Intuition: each pixel is allowed to change by up to a limit

Still in Digital Context

Perturbations

Page 14: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Adversarial Examples & !" NormPerturbations Impact to DNN

#$%& !" based Perturbation Method

Source Image YOLOv3 Detection

'( = ' − + , -./0(2!3--#,5('))Intuition: each pixel is allowed to change by up to a limit

Still in Digital Context

Perturbations

Page 15: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Adversarial Examples & !" NormPerturbations Impact to DNN

#$%A !" based Perturbation Method

Source Image Perturbations Perturbed Image

Intuition: # of pixels altered that matter the most

Still in Digital Context

Page 16: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Adversarial Examples & !" NormPerturbations Impact to DNN

#$%A !" based Perturbation Method

Source Image Perturbations YOLOv3 Detection

Intuition: # of pixels altered that matter the most

Still in Digital Context

Page 17: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Adversarial Examples & !" NormPerturbations Impact to DNN

Source Image Perturbed ImagePerturbations

Still in Digital Context

#$" !" based Perturbation Method Intuition: many small changes to many pixels %&'&%&() ∥ + − +- ∥""+ / 0 1(+-)

Page 18: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Adversarial Examples & !" NormPerturbations Impact to DNN

Source Image Perturbed ImagePerturbations

Still in Digital Context

#$" !" based Perturbation Method Intuition: many small changes to many pixels %&'&%&() ∥ + − +- ∥""+ / 0 1(+-)

Page 19: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Digital Perturbations Realistic Enough?

FGSM JSMA CW2

Alter pixel value of the

sky?

Alter pixel value of the

tree?Alter any

pixel value in the image?

Page 20: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Identify Opportunities by Completely Understanding YOLOv3 Inference Mechanism

Explore Chances of Physical White Box Attack against YOLOv3

Page 21: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Deep Dive into YOLOv3

YOLO v3

Object Detection Model

[147 Layers, 62M Parameters]

Input

[416x416x3]

Output

[10,647

Bounding

Boxes]

Image: http://media.nj.com/traffic_impact/photo/all-way-stop-sign-that-flashes-in-montclairjpg-30576ab330660eff.jpg

13x13

26x26

52x52

Page 22: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

• Common Objects in Context• 80 Classes: person, [car, truck, bus], [bicycle, motorcycle], [stop sign,

traffic light], etc.

Training Dataset – MS COCO Dataset

Page 23: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

car

0.01%

116x

90

156x

198373x326

Anchor

Boxes

!" = $ %" + '"!( = $ %( + '(!) = *)+,-

Center

Point

Object

Size !. = *.+,/

*.*)

13 x 13 Grid

('", '() = (11,2)

Prediction

Vector

Bounding Box 80 Class ConfidenceObjectness

%" %( %) %. *567 '8 '9 … … ':; '<=

■

stop sign

99%

×

car

0.01%

car

0.01%

YOLOv3 Prediction23

Page 24: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Threat Model : Physical Image Patch Attack

CompanyLogo Image Patches

Page 25: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Our Physical Attack Approach & Objectives

• Input Patch Construction• Differentiable to craft adversarial examples

• Attack Objectives• Make YOLOv3 detect fake object

• Make object disappear in front of YOLOv3

Page 26: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

CompanyLogo

Differentiable Input Patch Construction

Resize Perspective

Transformation

Page 27: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Our Physical Attack Approach & Objectives

• Input Patch Construction• Differentiable to craft adversarial examples

• Attack Objectives• Object Fabrication: make YOLOv3 detect fake object

• Object Vanishing: make object disappear in front of YOLOv3

Page 28: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Attack Objective 1 – Object Fabrication

A. Naive Fabrication• Push more detections towards a certain object

B. Precise Fabrication• Produce fake object at specific location

CompanyLogo

CompanyLogo

Page 29: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Attack Objective 2 – Object Vanishing

Make a certain object class disappear in the whole image.

Page 30: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

5 Inaccurate Patch Location

Challenges to the Success of Physical Attack

Controlled Perturbation Area1

2

Object appearance changesat various distances, angles

3

Various Light conditions:e.g. glaring, dimming

4 Color Distortion on various devices

Digital color palette32 x 21

Kyocera Taskalfa3551 ci

Captured by iPhoneXfrom a distance

Page 31: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Tactics to the Challenges:

• [Controlled Perturbation Area] Image-patch based Attack

• [Color Distortion] Color Management with the Non-Printability Loss (NPS)

• [Inaccurate Patch] Random Transformation (RT) during optimization iterations

• [Various Distances & Angles] RT + Total Variation regularization instead of Expectation-Over-Transformation

• [Various Light Condition] Get a stable environment

• More …

Page 32: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Color Management with Non-Printability Loss

Accessorize to a Crime: Real and Stealthy Attacks on State-Of-The-Art Face Recognition. Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael ReiterIn. In Proceedings of CCS 2016

Given ! ⊂ 0,1 & , a set of printable RGB triplets. '!( *̂ = ∏-∈/ |*̂ − *|For the perturbated 2, NPS(2) = ∑ 5- ∈ 6 '!((*̂) . NPS(2) ↓, color reproducibility ↑

No NPS With NPSPrinted&Captured by iPhone

Page 33: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Solution: Introduce Random Perspective Transformation

Random Transformation During Optimization Iterations

Generated Perturbation PatchPerfectly positioned?

!"#$%"&'(&

!"#$%"&'()

!"#$%"&'(*

Page 34: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

RT + TV for Various Distances & Angles

• Random Transformation + Total Variance Regulation�a different approach from EOTSimulate the transformations using RT + TV for various distances & angles instead of drawing from a distribution

Page 35: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Put Everything Together: An Iterative Optimization

NPS RT TV

!"#$%"&'(& !"#$%"&'() !"#$%"&'(*

Page 36: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)
Page 37: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Conclusion & Takeaway

• With careful setup, physical attacks are achievable against DNN-based object detection methods in a white box setting

• Defense is hard, a good safety and security metric has to be explored

• We call out efforts for a robust, adversarial example resistant model that is required in safety critical system like autonomous driving system

Page 38: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)

Scan Me


Mitigation of Adversarial Attacks through Embedded Feature ... · ing classifiers with embedded feature selection against adversarial examples crafted using different attack strategies

Adversarial Machine Learning —An Introductiondanadach/Security_Fall_19/aml.pdf · •Machine Learning (ML) •Adversarial ML •Attack •Taxonomy •Capability •Adversarial Training

Discriminator-Free Generative Adversarial Attack

Perception Deception: Physical Adversarial Attack Challenges and … · 2019. 12. 21. · Dr. Tao Wei Dr. Yunhan Jia Dr. Zhenyu(Edward) Zhong edwardzhong [at] baidu DOT com Weilin

Adversarial Ranking Attack and Defense...Adversarial Ranking Attack and Defense 5 Therefore, adversarial ranking attack is to nd a proper adversarial pertur-bation which leads the

NAttack: Learning the Distributions of Adversarial ... · transferable, meaning that one can design adversarial exam- ... We first review the NES based black-box adversarial attack

Hello false flags the art of deception in targeted attack attribution

Smoothing Adversarial Domain Attack and P-Memory ......Smoothing Adversarial Domain Attack and p-Memory Reconsolidation for Cross-Domain Person Re-Identification Guangcong Wang1,

Sparse Adversarial Attack via Perturbation FactorizationSparse Adversarial Attack via Perturbation Factorization 5 is another target label assigned by the attack, then it is called

Adversarial Laser Beam: Effective Physical-World Attack to DNNs in a Blink · 2021. 6. 11. · Adversarial Laser Beam: Effective Physical-World Attack to DNNs in a Blink Ranjie Duan1,2

Scratch that! An Evolution-based Adversarial Attack ... · Scratch that! An Evolution-based Adversarial Attack against Neural Networks Malhar Jere, Farinaz Koushanfar University of

Cyber Attack Intent Recognition and Active Deception using

Mitigation of Adversarial Attacks through Embedded …ing classifiers with embedded feature selection against adversarial examples crafted using different attack strategies. KEYWORDS

Tactics of Adversarial Attack on Deep Reinforcement Learning … · 2019-11-14 · Tactics of Adversarial Attack on Deep Reinforcement Learning Agents Yen-Chen Lin 1, Zhang-Wei Hong

TOWARDS FEATURE SPACE ADVERSARIAL ATTACK

1 Adversarial Attack and Defense on Graph Data: A Survey · arXiv:1812.10528v2 [cs.CR] 1 Apr 2020 1 Adversarial Attack and Defense on Graph Data: A Survey Lichao Sun, Yingtong Dou,

블랙박스 AI 기만공격기술 · 2020-07-14 · 목차 Adversarial Examples • 기만공격의목표/ 원리 기만공격분류 • Untargeted attack /Targeted attack • White-box

DPATCH: An Adversarial Patch Attack on Object Detectors · DPATCH: An Adversarial Patch Attack on Object Detectors Xin Liu 1, Huanrui Yang , Ziwei Liu2, Linghao Song , Hai Li , Yiran

MySQL Attack Mitigation Using Deception Technology

DECEPTION SOLUTION OVERVIEW - Attivo Networks · DECEPTION SOLUTION OVERVIEW DECEIVE. DETECT. DEFEND. DECEPTION FOR DETECTION AND ATTACK PATH VISIBILITY. ... high interaction deception

ShapeShifter: Robust Physical Adversarial Attack on Faster ... · ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detector 3 In this work, we propose ShapeShifter,

AGKD-BML: Defense Against Adversarial Attack by Attention

Exact Adversarial Attack to Image Captioning via Structured …openaccess.thecvf.com/content_CVPR_2019/papers/Xu_Exact... · 2019-06-10 · Exact Adversarial Attack to Image Captioning

Meta Gradient Adversarial Attack

WEB CACHE DECEPTION ATTACK - Black Hat Web cache functionality is set for the web application to cache static files ... Django: ... WEB CACHE DECEPTION ATTACK

1 Adversarial Attack on Large Scale Graph1 Adversarial Attack on Large Scale Graph Jintang Li, Tao Xie, Liang Chen*, Fenfang Xie, Xiangnan He, Zibin Zheng Abstract—Recent studies

Adversarial attacks on fingerprint liveness detection...based fingerprint liveness detection methods, including the-state-of-the-art one by adversarial attack technology. Sufficient

Learning to Attack: Adversarial Transformation Networks · Learning to Attack: Adversarial Transformation Networks Shumeet Baluja and Ian Fischer Google Research Google, Inc. Abstract

Detecting deception: adversarial problem solving in a low ...pages.stern.nyu.edu/.../fall2002/Detecting_Deception_CogSci2001.pdf · Detecting deception: adversarial problem solving

Chapter 3 Adversarial Attack · Chapter 3 Adversarial Attack Consider a data point x 0 2Rd belonging to class C i.Adversarial attack is a malicious attempt which tries to perturb

Detecting deception: adversarial problem solving in a low

Adversarial DefenseRecap Goodfellow et al, 2015 Untargeted Attack Recap Younis et al, 2019 Targeted Attack Outline Adversarial Defense Defense Strategies Adversarial Training …

Adversarial attack on deep learning · 2021. 6. 5. · • defense is hard against adversarial examples • adversarial examples transfer to different networks, trained on disjoint

Interpretable AI for Deep-Learning-Based Meteorological ... · Robustness to Adversarial Inputs Adversarial Examples for Four Attack Types Original DeepFool FGSM NewtonFool BIM Cat

Countering Attack Deception Techniques Pacsec04 - OUDOT Laurent [email protected]