![Page 1: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/1.jpg)
DEEP LEARNING QUALITY ASSURANCE
Marius Reuther, Founder DLQA
![Page 2: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/2.jpg)
SOCIAL MEDIA AND AI
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 2
https://www.chefblogger.me/2016/02/18/sind-frauen-
wirklich-so-auf-social-media/
https://techcrunch.com/wp-
content/uploads/2016/03/tay.jpg?w=1390&crop=1
![Page 3: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/3.jpg)
AI FAILING
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 3
https://techcrunch.com/wp-content/uploads/2016/03/tay.jpg?w=1390&crop=1
Tay.ai
• Microsoft‘s AI Chatbot
• Published in March 2016 on
• Emits salacious and abusive
tweets
• Shutdown only 16 hours after the
releasehttps://techcrunch.com/wp-
content/uploads/2016/03/tay.jpg?w=1390&crop=1
![Page 4: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/4.jpg)
Semantic Segmentation Road ModelObject Detection
CAMERA AI APPLICATIONS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 4
https://www.youtube.com/watch?v=_kxX09i4fds https://www.youtube.com/watch?v=PNzQ4PNZSzc https://www.youtube.com/watch?v=ETT34fn7OfE
![Page 5: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/5.jpg)
Collision PreventionObject Detection Semantic Segmentation
LIDAR AI APPLICATIONS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 5
https://www.youtube.com/watch?v=KdrHLXpYYlg https://www.youtube.com/watch?v=7NNpvtdrHkU https://www.youtube.com/watch?v=nXlqv_k4P8Q
![Page 6: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/6.jpg)
AI TrainingDataset
DATASET CREATION PROCESS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 6
Oil Rig Refinery
Collect DataAI Developer
Data
Labeling
![Page 7: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/7.jpg)
Datasets for Machine Learning in AD typically consist of 10k – 10 mio images.
Annotation may take 1h or more.
LABELING LANDSCAPE
Labeling Service Labeling ToolingTooling + Service
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 7
Computer Vision Annotation Tool
(CVAT)
![Page 8: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/8.jpg)
LabelMe
LABELME – OPEN SOURCE LABELING TOOL
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 8
![Page 9: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/9.jpg)
LabelMe
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 9
Car
car,
CAR,
Automobile,
Object,
Auto,
Vehicle,
Vehcl,
MotorizedVehicle,
motorized vehicle,
Motorized Vehicle,
…
LABELME – OPEN SOURCE LABELING TOOL
![Page 10: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/10.jpg)
LabelMe
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 10
Car
car,
CAR,
Automobile,
Object,
Auto,
Vehicle,
Vehcl,
MotorizedVehicle,
motorized vehicle,
Motorized Vehicle,
…
https://www.pinterest.it/pin/423479171186745584/
LABELME – OPEN SOURCE LABELING TOOL
![Page 11: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/11.jpg)
CVAT – OPEN SOURCE LABELING TOOLS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 11
CVAT
![Page 12: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/12.jpg)
Matlab Ground Truth Labeler Alp‘s Labeling Tool (ALT)
MORE OPEN SOURCE LABELING TOOLS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12
Overview about labeling tools:
https://de.slideshare.net/yuhuang/annotation-102542284
![Page 13: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/13.jpg)
OPEN SOURCE DATASETS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 13
Overview about open-source datasets:
https://de.slideshare.net/yuhuang/public-dataset-for-autonomous-driving
![Page 14: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/14.jpg)
TIME SPEND ON CREATING AI PROJECTS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 14
https://www.scribd.com/document/381514649/TRAIN-AI-2018-Andrej-Karpathy-Tesla(
datasets
models and algorithms
![Page 15: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/15.jpg)
GOOGLE PAPER ON MACHINE LEARNING & DATA
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 15
If data replaces code in ML
systems, […] then […] testing
of input data is critical to a
well-functioning system.
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
![Page 16: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/16.jpg)
FREQUENT QUALITY ISSUES
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 16
http://blogs.yis.ac.jp/20hepburng/2016/05/24/english-unit-3-reflection-perspective-and-prejudice-english/
![Page 17: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/17.jpg)
FREQUENT QUALITY ISSUES
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 17
Note:
An unbiased and labeling-
independent QA workflow
helps to improve the overall
dataset quality while
keeping track on the
requirements, latest updates
and doubtful situations as
well as insufficiencies!
![Page 18: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/18.jpg)
AI TrainingDataset
DATASET CREATION PROCESS (WITH QA)
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 18
Oil Rig Refinery
Collect Data
Data
Labeling
Quality
Assessment
Inspection
|✓
AI Developer
![Page 19: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/19.jpg)
AI TrainingDataset
DATASET CREATION PROCESS (WITH QA)
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 19
Oil Rig Refinery
Collect Data
Data
Labeling
Quality
Assessment
Inspection
|✓
AI Developer
![Page 20: DEEP LEARNING QUALITY ASSURANCE - autonomous driving · AI FAILING MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 12/20/2018 3](https://reader030.vdocument.in/reader030/viewer/2022040409/5ec7f3b0714be464251822fd/html5/thumbnails/20.jpg)
THANKS
12/20/2018MARIUS REUTHER, DLQA @ AUTONOMOUS DRIVING MEETUP MUNICH 20
Marius Reuther, Founder DLQA
DEEP LEARNING QUALITY ASSURANCE
www.dlqa.de