s9554 - fast training of deep neural networks using brain ... · s9554 - fast training of deep...
TRANSCRIPT
S9554 - Fast Training of
Deep Neural Networks Using
Brain-Generated Labels
Sergey Vaisman, VP R&D, InnerEye
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 2
“Without humans, artificial intelligence is stillpretty stupid..”
Image: The New York Times
(The Wall Street Journal)
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 3
Data
Collection
Dataset
Annotation
Neural
Network
Design
Deployment
New Data
Neural
Network
Training
Optimization
Dataset
Exploration
AI Training Challenges
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 4
0.4
0.5
0.6
0.7
0.8
0.9
1
0 20 40 60 80 100 120 140 160 180 200 220 240 260
AU
C(A
rea
Un
der
th
e C
urv
e)
Time [min]
Classification Performance
Brain In The Loop - Iterative AI Training Framework
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 5
Image Classification Average Performance
10x times faster
Trained on NVIDIA GeForce GTX1080 ti GPU
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 6
InnerEye – The CompanyCombining Human Intelligence with Artificial Intelligence
▪ Founded in 2014 - Technology spin-off from Israel’s The Hebrew and Ben-Gurion Universities
▪ Offices in Herzliya, Israel and Tokyo, Japan
▪ Over $6M of funding provided so far
▪ Products: Visual content review, AI Training and Validation, Connected Human
▪ Management Team:
Uri AntmanCEO
Prof. Amir B. GevaFounder and CTO
Prof. Leon Y. DeouellFounder and CSO
Sergey VaismanVP R&D
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 7
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye
Agenda
▪ Iterative AI Training Framework
▪ Performance
▪ Use Cases
8
Images
Images for human review
Brainwaves Classification Network
Image Classification Network
Trained
Network
InnerEye AI Training Framework
Soft Labels
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 9
Convergence
Criterion
InnerEye AI Training Framework
Images
Images for human review
Convergence Criterion
Image Classification Network
Trained Network
Deployment
Brainwaves Classification
NetworkSoft Labels
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye10
Visual Processing
Response Planning
Motor Execution
Distraction caused by
motor response
Sensation
ImageUser
Response
Decision Making
Brain activity measurement
Tapping Into The Brain
InnerEye Bypass
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 11
Brain Activity Measurement - EEG
t = 0.33s t = 0.66s t = 1s t = 1.33s t = 1.66s t = 2st = 0
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 12
t = 0.33s t = 0.66s t = 1s t = 1.33s t = 1.66s t = 2st = 0
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 13
t = 0.33s t = 0.66s t = 1st = 0
Single Trial Spatio-Temporal Activity
time (ms)
Tria
l #
0 200 400 600 800
1000
2000
3000
4000
5000
6000
7000
8000
9000
T Time samplesx D Channels
Single Trial Spatio-Temporal Matrix
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 14
t = 1st = 0
Single Trial ClassificationTargets
time (ms)
Tria
l #
0 200 400 600 800
100
200
300
400
500
600
Non Targets
time (ms)
Tria
l #
0 200 400 600 800
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 100 200 300 400 500 600 700 800 900-5
0
5
10
time (ms)
mV
PzNon target
Target
t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms t = 200ms t = 300ms t = 450ms t = 550ms t = 700ms
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 15
InnerEye Brainwaves Classification Network
Deep Neural Network Ensemble
T Time samplesx D Channels
INPUT:Single Trial
Spatio-Temporal Matrix
OUTPUT:Trial
Classification score
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 16
Brainwaves Classification Scores DistributionScore
Sports CarsRegular Cars
Image
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 17
Soft Labels Concept▪ These images contain flowers but not all of them should contribute equally to the
learning process
▪ Can we create more informative labels to address the diversity and improve classification accuracy?
Label: FLOWER Label: FLOWER Label: FLOWER
Image Source: Google Open Images Dataset
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 18
EEG Classification Scores Are Used to Generate Soft Labels
▪ Images that received high or low scores from the EEG classifier are given higher weight
▪ Images that received intermediate (inconclusive) scores from the EEG classifier are given lower weight
𝑤𝑖 = ቊ𝑐𝑖 , 𝑦𝑖 = 1
1 − 𝑐𝑖 , 𝑦𝑖 = 0
𝑐𝑖 = EEG Classification Score
𝑦𝑖 = Sample Label
𝑤𝑖 = Sample Weight (Soft Label)
THR = Classification Threshold
𝑦𝑖 = ቊ1, 𝑐𝑖 ≥ 𝑇𝐻𝑅0, 𝑐𝑖 < 𝑇𝐻𝑅
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 19
Soft Labels Are Correlated with Human Confidence Level
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 20
InnerEye AI Training Framework
Images
Images for human review
Convergence Criterion
Image Classification NetworkBrainwaves
Classification Network
Deployment
Soft Labels
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 21
Image Classification Network▪ In each iteration, image classification layers are trained using the new
generated soft labels64 filters
X 2 blocks 128 filters
X 2 blocks 256 filters
X 3 blocks512 filters
X 3 blocks 512 filters
X 3 blocks 32
neurons
Sigmoid
output
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 22
Soft Labels Are Used As Sample Weights in Loss Function
▪ We add sample weights to the cross-entropy loss function:
𝐿(𝑥𝑖) = Cross Entropy Loss Function
𝑥𝑖 = Sample
𝑦𝑖 = Sample Label
𝑤𝑖 = Sample Weight (Soft Label)
𝑝𝑖 = Sample Prediction
𝐿 𝑥𝑖 = −𝒘𝒊 𝑦𝑖𝑙𝑜𝑔 𝑝𝑖 + 1 − 𝑦𝑖 𝑙𝑜𝑔(1 − 𝑝𝑖)
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 23
▪ The learning algorithm selects new samples to learn from in the next iteration based on Confidence criterion:
▪ Only the least confident samples (𝐶𝑖 < THR) will be sent for human review
▪ Also used as Convergence Condition
Convergence and Active Learning Iterations
𝐶𝑖 = max𝑗
𝑃 𝑦𝑖 = 𝑗|𝑥𝑖
𝐶𝑖 = Confidence for Sample i
j = Class index
𝑦𝑖 = Predicted Sample label
𝑥𝑖 = Sample i
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 24
Soft Labels Improve Neural Network Performance
(*) AUC shown after the first iteration
(permuted sample weights of the labels)
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 25
Active Learning Combined With Soft Labels Improves Neural Network Performance
(*) AUC shown after the first iteration
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 26
Images
Images for human review
Brainwaves Classification Network
Image Classification Network
Trained
Network
InnerEye AI Training Framework
Soft Labels
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye
Convergence
Criterion
27
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye
Agenda
▪ Iterative AI Training Framework
▪ Performance
▪ Use Cases
28
We started with 2,800 Cars images to be classified. Initial
network performance for Sports cars detection: AUC=0.5
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 29
After Iteration 1 (T=4.2 min): Human expert reviewed
200 images. Network performance: AUC=0.8
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 30
After Iteration 2 (T=8.7 min): Human expert reviewed
400 images. Network performance: AUC=0.86
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 31
After Iteration 3 (T=13.3 min): Human expert reviewed
589 images. Network performance: AUC=0.88
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 32
After Iteration 4 (T=17.2 min): Human expert reviewed
718 images. Network performance: AUC=0.9
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 33
After Iteration 5 (T=20.6 min): Human expert reviewed
795 images. Network performance: AUC=0.92
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 34
After 20.6 minutes the network is trained to detect Sports Cars.
The human expert was required to review only 28% of the images
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 35
Examples of correctly classified sports cars:Images with the highest score from the InnerEye system trained to classify Sport Cars:
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 36
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 37
Image Classification Performance vs. % of Training Data
38
(*) AUC shown after the first iteration
Subject 1 Subject 2 Subject 3
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 38
Image Classification Performance vs. Training Time
(*) Trained on NVIDIA GeForce GTX 1080 Ti
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
0 25 50 75 100 125 150 175 200 225 250 275
AU
C
Time [min]
Subject 1
InnerEye Training Method
Manual Training Method
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
0 25 50 75 100 125 150 175 200 225 250 275
AU
C
Time [min]
Subject 2
InnerEye Training Method
Manual Training Method
0.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
0 25 50 75 100 125 150 175 200 225 250 275
AU
C
Time [min]
Subject 3
InnerEye Training Method
Manual Training Method
▪ Time savings come from combination of fast presentation rate and faster convergence
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 39
Image Classification Average Performance
10x times faster
Trained on NVIDIA GeForce GTX1080 ti GPU
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 40
Soft Labels Compensate for EEG Misclassified Samples Correctly classified sports cars
Weight: 0.93 Weight: 0.81 Weight: 0.87
False Positives
Weight: 0.26Weight: 0.3 Weight:0.22
Weight: 0.4 Weight: 0.43 Weight: 0.31
Missed sports cars
Weight: 0.85 Weight: 0.79 Weight: 0.91
Correctly classified regular cars
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 41
Soft Labels Compensate for EEG Misclassified Samples
(*) AUC shown after the first iteration
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 42
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye
Agenda
▪ Iterative AI Training Framework
▪ Performance
▪ Use Cases
43
Multiclass ClassificationAccuracy: 0.81
Event Related Potentials measured on electrode T5
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 44
Validation of Annotation Quality
▪Fast Screening and amendment of low confidence annotated data
Screenshot of the output from InnerEye system output of detecting images wrongly labeled as “flowers”
Source: Google Open Images Dataset
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 45
Combined Brain-Computer Visual Network
IMG Net
EEG Net
Score: Classification:
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 46
InnerEye Driver
Car SensorsRoad Conditions
Environment:
Driving Agent
State Reward Action
Screenshot of the output from “InnerEye driver” brain responding to seeing pedestrian crossing the road, measuring Hazard Detection, Attention and Emotion
Reinforcement Learning for Autonomous Driving▪ Incorporating brain insights in the training process of the AI driving agent
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 47
Efficiency & Throughput
Cost
PersonalizationReal-Time Interface
Privacy
Summary
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 48
THANK YOU!COME SEE OUR DEMO
AT BOOTH 335
Contact me at: [email protected]
S9554 - Fast Tra in ing of Deep Neural Networks Using Brain-Generated Labels Sergey Vaisman, InnerEye 49