systematic analysis, testing, and improvement of cpsmlh. niederreiter, “random number generation...

Systematic Analysis, Testing, and Improvement of CPSML

Tommaso Dreossi

1

Joint work with:Daniel Fremont, Shromona Ghosh, Xiangyu Yue, Alexandre Donze

Kurt Keutzer, Alberto Sangiovanni-Vincentelli, Sanjit A. Seshia

UC Berkeley

Cyber-Physical Systems (CPS)Integration of computation with physical processes

Building systems Factory automation Automotive

Smart citiesPower generation Avionics 2

Cyber-Physical Systems + ML/AI (CPSML)Growing use of Machine Learning/AI in CPS

20152010200520001995

200

400

StartupsdevelopingAIsystems

600

Year

Activestartups

2015201020052000Year

5K

10K

15K

Pape

rs

AnnuallypublishedAIpapers

Source:AIindex.org,Scopus,Crunchbase,VentureSource,SandHillsEconometrics 3

Cyber-Physical Systems + ML/AI (CPSML)Growing use of Machine Learning/AI in CPS

Many safety-critical applications

Source:DMVCA

0 5 10 15 20

Recklesslybehavingroaduser

Incorrectbehaviorprediction

Softwarediscrepancy

Hardwarediscrepancy

Perceptiondiscrepancy

Unwantedmaneuver

Waymo disengagementreportCalifornia,2017

Disengagement4

Formal methods approach

𝑆 ∥ 𝐸 ⊨ 𝜑SystemS

EnvironmentESpecification𝜑

Yes(proof)

No(counterexample)

Challenges for Verified AI

5

Challenges for Verified AIFormal methods approach



Yes(proof)

No(counterexample)

• Complexmodels• E.g.,AlexNet,60Mparameters,650Kneurons)

• Largeinputspaces• E.g.,KITTI images:256^(1392x512x3)

NeednewmethodsforAbstraction andModularReasoning6




Yes(proof)

No(counterexample)

• Interactionwithcomplexenvironments/agents

Needforrepresentingenvironmentscenarios7




Yes(proof)

No(counterexample)

• Howdoyouformalizeperceptiontasks?

Needfornewspecificationformalisms8




Yes(proof)

No(counterexample)

Ourapproach:• System:

• CompositionalanalysisofCPS-ML• AbstractionofMLmodulesinputspace

• Environment• Scenic– Scenariodescriptionlanguage

• Specification• System-levelspecifications 9

Outline

1. RunningCPSML example– Automaticemergencybrakingsystem2. Specification

• System- vsModule-levelspecification3. System

• Compositionalfalsification• MLinputabstraction• Counterexample-guidedaugmentation

4. Environment• Scenic:Scenariodescriptionlanguage

5. Conclusion

10

CPSML ExampleAutomatic Emergency Braking System (AEBS)

Environment

ControllerPlant

ML perception

• Goal: brake when an obstacle is near • Challenges:

• How to explore distance/velocity?

• How to analyze images?

• How to combine distance, velocity, images?

distance, velocity

11

SpecificationSystem- vs Module-level Specification

Environment

ControllerPlant

ML perception

distance, velocity

• Goal: brake when an obstacle is near• Specifications:

• “Never collide” (distance > 0)• “Correctly detect obstacles” 12

Outline





5. Conclusion

13

CPSML input space intractable• Idea: focus on meaningful CPS+ML input

combinations• Intuition: “If car is far, misclassification won’t

affect our system”

Strategy:1. Analyze CPS gathering info on ML role2. Use collected info to target ML3. Compose CPS + ML narrowed input spaces4. Perform targeted falsification

Compositional Falsification

Environment

ControllerPlant

ML perception

14Dreossiet.al,CompositionalFalsificationofCyber-PhysicalSystemswithMachineLearningComponents,NFM 2017

Compositional Falsification• Identifying regions of interest for AEBS• Perform optimistic/pessimistic analyses of NN

MLcorrect MLwrong Potentiallyunsaferegion(dependingonML)

15Dreossiet.al,CompositionalFalsificationofCyber-PhysicalSystemswithMachineLearningComponents,NFM 2017

ML Analyzer

• How analyze ML feature space?• E.g., image classifier: a lot of pictures to analyze

• Idea: Focus on semantic alterations

✕ ✓✓

Plausiblealterations16Dreossiet.al,SystematicTestingofConvolutionalNeuralNetworksforAutonomousDriving,RMLW 2017

brightness car z-pos

car x-posModification space

Modificationspace

✓

Neural network𝑦 ∈ {𝑐𝑎𝑟, ¬𝑐𝑎𝑟}

✓ ✕

✕

✕

✓

✓

✓

✓

✕

ML AnalyzerSystematically analyze modifications of interest

Picture spaceSystematicsampling

17

ML AnalyzerSampling methods

H.Niederreiter,“RandomNumberGenerationandQuasi-Monte-CarloMethods”,1992R.Y.Rubinstein etal.,“TheCross-EntropyMethod,AUnifiedApproachtoCombinatorialOptimization,Monte-CarloSimulation,andMachineLearning”,200418

Method Sampling speed Diversity Counterexamplefinding

Uniform random ✓ ✕ ✕

Uniform random + distance constraint ✕ − ✕

Low-discrepancy ✕ ✓ −

Cross entropy ✕ ✕ ✓

Sample Results

Inception-v3 Neural Network(pre-trained on ImageNet using TensorFlow)

This misclassificationmay not be of concern But this one

is a real hazard

Corner caseMisclassification

cluster

AEBS

19

Sample Results

Example of counterexamples

squeezeDet(trained on synthetic images)

Blind spot

squeezeDet

20

• Whattodowiththegeneratedcounterexamples?1. Analyzethemandprovideexplanations(errortables)2. Augmenttrainingsets

Misclassifications

Id Carcolor Background Orientation

1 Red Countryside Front

2 Orange Forest Back

3 White Forest Front

4 Green Forest Back

Counterexample-guided augmentation

Train Test Test Aug

Error table

Counterexamples 21Dreossiet.al,Counterexample-GuidedDataAugmentation,IJCAI2018

Sampler Imagegenerator Modelf Counter

example?

Yes

No

𝕄 m x ŷ

1. Growaugmentationset 𝔸2. Updateerrortable

𝔸 largeenough?

No

Yes

Start

End

Id Carcolor Background Orientation

1 Red Countryside Front

2 Orange Forest Back

3 White Forest Front

4 Green Forest Back

22

FindcounterexamplesandaugmenttrainingsetCounterexample-guided augmentation

Dreossiet.al,Counterexample-GuidedDataAugmentation,IJCAI2018

AugmentationComparisonAugmentation

Train - 1.5k Test - 0.75k

Counterexamples

Samplingmethodscomparison

Model Precision Recall t (sec)Original .61 .75Standard augmentation .69 .80Uniform random .76 .87 ~30Constrain .75 .86 ~92Low-discrepancy .79 .87 ~55Cross-entropy .78 .78 ~70

Test - 0.75k Aug - 0.75k

23

Outline





5. Conclusion

24

Environment Description

Problem• Largeandunstructuredinputspace• Generatemeaningfulscenes(fortestingortraining)

Car Model Car Location Car Orientation

Number of Cars Reference Scene Background

Car Color Weather Time of Day

Idea: Use simulators to model environment (e.g., GTAV)

25

• Scenic: probabilistic programming language defining distributions over scenes

• Example: a badly parked car

ScenicA Scenario Description Language

26Fremontet.al,Scenic:Language-BasedSceneGeneration,underreview

Scenic ApplicationsTesting

Exploring the behavior of the system under different conditions:

Bright and clear weather Dark and rainy weather

27

Scenic ApplicationsTraining

Generate hard cases, e.g., one car partially occluding another:

28

Scenic ApplicationsReasoning

29


30


Scenic makes it easy to generalize along different dimensions:

Add noise Change car model Change global position

31

Conclusion

32

Futurework• Mixreal/syntheticdata• Domainadaptation/randomization• Morecomplexdata:lidar,radar,etc.

Summary• Frameworkforsystem-levelcounterexamples• CNNanalyzer(simulationbased)• Counter-exampleguidedaugmentation• Scenic:Scenariodescriptionlanguage

systematic analysis, testing, and improvement of cpsmlh. niederreiter, “random number generation...

Documents