testing for autonomous driving - testing deep neuronal ... · motivation basics testing deeptest...
TRANSCRIPT
![Page 1: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/1.jpg)
Testing for autonomous drivingTesting Deep Neuronal Networks
Jakob Krause
Freie Universität Berlin, Department of Mathematics and Computer Science,Institute of Computer Science, Software Engineering Group
May 8, 2019
Jakob KrauseTesting DNNs, May 8, 2019
![Page 2: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/2.jpg)
OutlineMotivation Basics Testing Deep Test DeepRoad Resume References
1 Motivation
2 Basics
3 Testing
4 Deep Test
5 DeepRoad
6 Resume
Jakob KrauseTesting DNNs, May 8, 2019
![Page 3: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/3.jpg)
IntroductionMotivation Basics Testing Deep Test DeepRoad Resume References
▶ significant progress in machine learning (ML) did lead to safetycritical systems like autonomous cars
▶ companies like Google (Waymo), Tesla, BMW are building andtesting those cars
▶ even with the big progress and money in the field the industryalready produced some big accidents
Jakob KrauseTesting DNNs, May 8, 2019
1
![Page 4: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/4.jpg)
What can go wrong?Motivation Basics Testing Deep Test DeepRoad Resume References
Date Cause OutcomeHyundai 2014 Rain fall Crashed while testingTesla 2016 Image contrast Killed the driverGoogle 2016 Failed to estimate speed Hit a bus
Uber 2018 too slow detectionno emergency brakes Killed pedestrian
Table 1: Example of real-world accidents involving autonomous cars
=> Most of these accidents happen in rare corner cases!
Jakob KrauseTesting DNNs, May 8, 2019
2
![Page 5: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/5.jpg)
Basic overview about neural networksMotivation Basics Testing Deep Test DeepRoad Resume References
▶ a graph with weighted edges (nodes are called neurons)▶ nodes contain non linear functions(activation function)▶ receives an input and returns a value (f.I. classification)▶ ’trained’ with labeled data▶ tested with labeled data different to training set
Jakob KrauseTesting DNNs, May 8, 2019
3
![Page 6: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/6.jpg)
What is a neural networkMotivation Basics Testing Deep Test DeepRoad Resume References
Figure 1: Simple neural network
Jakob KrauseTesting DNNs, May 8, 2019
4
![Page 7: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/7.jpg)
What is a deep neural networkMotivation Basics Testing Deep Test DeepRoad Resume References
Figure 2: Deep neural network
Jakob KrauseTesting DNNs, May 8, 2019
5
![Page 8: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/8.jpg)
What is an autonomous car?Motivation Basics Testing Deep Test DeepRoad Resume References
Figure 3: Simple autonomous car
Jakob KrauseTesting DNNs, May 8, 2019
6
![Page 9: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/9.jpg)
Testing DNNsMotivation Basics Testing Deep Test DeepRoad Resume References
▶ On a conceptual level the error prone corner case a comparableto logic bugs in regular software
▶ Similar to a bug detection and patching cycle in traditionalsoftware, detected bugs are fixed by adding error inducingtraining data
Jakob KrauseTesting DNNs, May 8, 2019
7
![Page 10: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/10.jpg)
Why is it difficult to test DNNsMotivation Basics Testing Deep Test DeepRoad Resume References
▶ logic of a DNN is expressed by neurons and weights instead ofcontrol statements (traditional software)
▶ definition and image space is hughe▶ finding inputs that will result in high model coverage in a DNN is
significantly because of non-linearity▶ manually creating specifications for complex DNN systems like
autonomous cars is infeasible as the logic is too complex tomanually encode as it involves mimicking the logic of a humandriver
Jakob KrauseTesting DNNs, May 8, 2019
8
![Page 11: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/11.jpg)
Limitations of existing DNN testingMotivation Basics Testing Deep Test DeepRoad Resume References
▶ expensive labeling effort▶ often data needs to labeled by hand
▶ low test coverage▶ no systematic approach for covering different rules of the
network▶ Example : Splitting data sets in training set and testing sets▶ => no guarantee testing set will test all learned rules
Jakob KrauseTesting DNNs, May 8, 2019
9
![Page 12: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/12.jpg)
Problem of low-coverage DNN testsMotivation Basics Testing Deep Test DeepRoad Resume References
Figure 4: Flow of traditional code compared to NN
Jakob KrauseTesting DNNs, May 8, 2019
10
![Page 13: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/13.jpg)
Deep TestMotivation Basics Testing Deep Test DeepRoad Resume References
▶ testing methodology for automatically detecting erroneousbehaviors of DNN-based software of self-driving cars
▶ leverage the notion of neuron coverage (i.e., the number ofneurons activated by a set of test inputs) to systematicallyexplore different parts of the DNN logic
▶ empirically demonstrate that changes in neuron coverage arestatistically correlated with changes in the actions of self-drivingcars (e.g., steering angle)
▶ demonstrate that different image transformations that mimicreal-world differences in driving conditions like changingcontrast/brightness, rotation of the camera result in activation ofdifferent sets of neurons in the self-driving car DNNs
Jakob KrauseTesting DNNs, May 8, 2019
11
![Page 14: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/14.jpg)
Systematic Testing with Neuron CoverageMotivation Basics Testing Deep Test DeepRoad Resume References
▶ IO-space is big for autonomous vehicles −→ we need a partition
Neuron Coverage
Neuron Coverage =|Activated Neurons|
|Total Neurons|
Assumption 1All inputs with similar Neuron Coverage have similar behaviour
Jakob KrauseTesting DNNs, May 8, 2019
12
![Page 15: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/15.jpg)
Increasing Coverage with Synthetic ImagesMotivation Basics Testing Deep Test DeepRoad Resume References
▶ generated inputs need to be realistic▶ applying transformations to seed images
Figure 5: Example of some applied transformations
Jakob KrauseTesting DNNs, May 8, 2019
13
![Page 16: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/16.jpg)
Combining Transformations to Increase CoverageMotivation Basics Testing Deep Test DeepRoad Resume References
▶ using multiple transformations to increase coverage▶ space of applied transformations is very big −→ heuristic
Jakob KrauseTesting DNNs, May 8, 2019
14
![Page 17: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/17.jpg)
Test Oracle with Metamorphic RelationsMotivation Basics Testing Deep Test DeepRoad Resume References
▶ Metamorphic testing (MT) is a property-based software testingtechnique, which can be an effective approach for addressing thetest oracle problem and test case generation problem
▶ Simple example sin(π − x) = sin(x)▶ define relationships between the car’s behaviors across certain
types of transformations▶ f.i. steering angle should not change significantly under changing
weather conditions▶ steeringAngle(Imgoriginal) = steeringAngle(Imgsynthetic)
Jakob KrauseTesting DNNs, May 8, 2019
15
![Page 18: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/18.jpg)
Test Oracle with Metamorphic RelationsMotivation Basics Testing Deep Test DeepRoad Resume References
▶ steering angle can be a little different but still correct▶ trade-of in being more strict with more false positive and visa
versa▶ angle for original images Θoi
▶ angle for transformed images Θti
▶ angle for manual labels Θ̂i
▶ MSEorig = 1n∑N
i=1(Θ̂i −Θoi)2
Metamorphic relation
(Θ̂i −Θti)2 ≤ λMSEorig
Jakob KrauseTesting DNNs, May 8, 2019
16
![Page 19: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/19.jpg)
ImplementationMotivation Basics Testing Deep Test DeepRoad Resume References
▶ How do we benchmark our proposal▶ picked 3 strong AIs based on the Keras Framework from the
Udacity Autonomous driving challenge▶ lets see if we can error and measure there behaviour
Jakob KrauseTesting DNNs, May 8, 2019
17
![Page 20: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/20.jpg)
ResultsMotivation Basics Testing Deep Test DeepRoad Resume References
Research question 1Do different input-output pairs result in different neuron coverage ?
▶ Steering angel: neuron coverage correlates with statisticalsignificance
▶ Steering direction: neuron coverage varies with statisticalsignificance
Result 1Neuron coverage is correlated with input-output diversity and canbe used to systematic test generation
Jakob KrauseTesting DNNs, May 8, 2019
18
![Page 21: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/21.jpg)
ResultsMotivation Basics Testing Deep Test DeepRoad Resume References
Research question 2Do different realistic image transformations activate differentneurons ?
Result 2Different image transformations tend to activate different sets ofneurons
Jakob KrauseTesting DNNs, May 8, 2019
19
![Page 22: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/22.jpg)
ResultsMotivation Basics Testing Deep Test DeepRoad Resume References
Research question 3Can neuron coverage be further increased by combining differentimage transformations
Result 3By systematically combining different image transformations,neuron coverage can be improved by around 100% w.r.t. thecoverage achieved by the original seed images.
Jakob KrauseTesting DNNs, May 8, 2019
20
![Page 23: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/23.jpg)
ResultsMotivation Basics Testing Deep Test DeepRoad Resume References
Research question 4Can we automatically detect erroneous behaviors usingmetamorphic relations?
Result 4With neuron coverage guided synthesized images, DeepTestsuccessfully detects more than 1,000 erroneous behavior aspredicted by the three models with low false positives.
Jakob KrauseTesting DNNs, May 8, 2019
21
![Page 24: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/24.jpg)
ResultsMotivation Basics Testing Deep Test DeepRoad Resume References
Research question 5Can retraining DNNs with synthetic images improve accuracy?
Result 5Accuracy of a DNN can be improved up to 46% by retraining theDNN with synthetic data generated by DeepTest
Jakob KrauseTesting DNNs, May 8, 2019
22
![Page 25: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/25.jpg)
ConclusionMotivation Basics Testing Deep Test DeepRoad Resume References
▶ tool for automated testing of DNN-driven autonomous cars▶ maximizing neuron coverage with synthetic test images by
realistic transformations▶ using domain typical metamorphic relations to find erroneous
behaviours without detailed specification
Jakob KrauseTesting DNNs, May 8, 2019
23
![Page 26: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/26.jpg)
Problems with this Deep TestMotivation Basics Testing Deep Test DeepRoad Resume References
▶ change of camera lenses are not considered f.i when its raining▶ transformations are not very realistic▶ no complex scenes (like snowy scenes)
Figure 6: Unrealistic Transformations
Jakob KrauseTesting DNNs, May 8, 2019
24
![Page 27: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/27.jpg)
DeepRoadMotivation Basics Testing Deep Test DeepRoad Resume References
▶ GAN-Based Metamorphic Testing and Input ValidationFramework for Autonomous Driving Systems
▶ introduces input validation▶ also uses metamorphic relationships▶ technical paper▶ improves synthetic images processing by using generative
adversarial networks
Jakob KrauseTesting DNNs, May 8, 2019
25
![Page 28: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/28.jpg)
Input validationMotivation Basics Testing Deep Test DeepRoad Resume References
▶ in classical software we found to catch malformed data. Forexample a webserver wants to stop processing when the json ismalformed
▶ NN would be much safer if there suddenly can tell on a snowyroad that the cant process this environment
▶ any 640x480 RGB picture −→ to weak▶ only data from training set −→ to strong
minj||h(i)− h(j)||2 < Θ
Jakob KrauseTesting DNNs, May 8, 2019
26
![Page 29: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/29.jpg)
How does the improved image generation workMotivation Basics Testing Deep Test DeepRoad Resume References
Figure 7: Transformations Structure
▶ x1 and x2 same environemnt with different weather (sunny andrainy), Si Domains (sunny and rainy), Z space where x1 and x2are same, Ei encoders to a space Z , Gi generators
Jakob KrauseTesting DNNs, May 8, 2019
27
![Page 30: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/30.jpg)
Improved sythetical images via GANMotivation Basics Testing Deep Test DeepRoad Resume References
Figure 8: GAN Transformations
Jakob KrauseTesting DNNs, May 8, 2019
28
![Page 31: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/31.jpg)
ConclusionMotivation Basics Testing Deep Test DeepRoad Resume References
▶ proposed an unsupervised learning framework to synthesizerealistic driving scenes to test inconsistent behaviors ofDNN-based autonomous driving systems
▶ successfully detect thousands of inconsistent behavior
Jakob KrauseTesting DNNs, May 8, 2019
29
![Page 32: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/32.jpg)
ResumeMotivation Basics Testing Deep Test DeepRoad Resume References
▶ We have seen 2 approaches for testing deep neuronal networks▶ Neuron coverage as a metric of test coverage▶ Metamorphic relations as a test oracle▶ Synthetic image generation to increase coverage and safety▶ concept of input validation for DNNs
Jakob KrauseTesting DNNs, May 8, 2019
30
![Page 33: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/33.jpg)
Questions?Motivation Basics Testing Deep Test DeepRoad Resume References
Jakob KrauseTesting DNNs, May 8, 2019
31
![Page 34: Testing for autonomous driving - Testing Deep Neuronal ... · Motivation Basics Testing DeepTest DeepRoad Resume References 1Motivation 2Basics 3Testing 4DeepTest 5DeepRoad 6Resume](https://reader034.vdocument.in/reader034/viewer/2022050119/5f4fdc3a61a8d86f5c5ce767/html5/thumbnails/34.jpg)
References IMotivation Basics Testing Deep Test DeepRoad Resume References
[1] Yuchi Tian et al. “DeepTest: Automated Testing ofDeep-neural-network-driven Autonomous Cars”. In:Proceedings of the 40th International Conference on SoftwareEngineering. ICSE ’18. Gothenburg, Sweden: ACM, 2018,pp. 303–314. DOI: 10.1145/3180155.3180220. URL:http://doi.acm.org/10.1145/3180155.3180220.
[2] Mengshi Zhang et al. “DeepRoad: GAN-based MetamorphicAutonomous Driving System Testing”. In: CoRRabs/1802.02295 (2018). arXiv: 1802.02295. URL:http://arxiv.org/abs/1802.02295.
Jakob KrauseTesting DNNs, May 8, 2019
32