cheap, fast, and low power deep learning: i need it now ...of deep learning models ... estimation -...
Post on 07-Jul-2020
0 Views
Preview:
TRANSCRIPT
NeurIPS/VIPER December 13, 2019 1
Cheap, Fast, and Low Power Deep Learning: I need it now!(Please and Thank You)
Edward J. Delp
Video and Image Processing Laboratory (VIPER)School of Electrical and Computer Engineering
School of Biomedical Engineering Purdue University
West Lafayette, Indiana
email: ace@ecn.purdue.eduhttps://engineering.purdue.edu/~ace/
NeurIPS/VIPER December 13, 2019 2
Video and Image Processing Laboratory(VIPER)
www.viperlab.org
NeurIPS/VIPER December 13, 2019 3
Acknowledgements
NeurIPS/VIPER December 13, 2019 4
NeurIPS/VIPER December 13, 2019 5
Outline
• Introduction of Low Power Deep Learning
• Applications That Need Low Power Deep Learning
– Health care monitoring
– Biomedical image analysis
– Image Based Phenotyping
NeurIPS/VIPER December 13, 2019 6
Deep Learning “Requirements”
Reference: S. Bianco, et al., Benchmark Analysis of Representative Deep Neural Network Architectures, IEEE Access, 2018
NeurIPS/VIPER December 13, 2019 7
Low Power Deep Learning
• Current deep learning models are both power and memory intensive
• Need more analysis of computational cost (memory usage, inference time)
NeurIPS/VIPER December 13, 2019 8
After training...
NeurIPS/VIPER December 13, 2019 9
Images Per Second vs. Accuracy
Reference: S. Bianco, et al., Benchmark Analysis of Representative Deep Neural Network Architectures, IEEE Access, 2018
• Less computation hurts the performance
NeurIPS/VIPER December 13, 2019 10
Inference Speed on Embedded System
Nvidia Titan X with 3840 cores(Power consumption is approximately 250w)
Nvidia Jetson TX1 board with 256 cores(Power consumption is approximately 10w)
Reference: S. Bianco, et al., Benchmark Analysis of Representative Deep Neural Network Architectures, IEEE Access, 2018
NeurIPS/VIPER December 13, 2019 11
Inference Speed on Non-GPU Device
Reference: D. Velasco-Montero, et al., Performance analysis of real-time DNN inference on Raspberry Pi, SPIE Real-Time Image and Video Processing 2018
• Frame per second on Raspberry Pi• Power consumption is approximately 6w
NeurIPS/VIPER December 13, 2019 12
Methods for Reducing the Complexity of Deep Learning Models
• Pruning– Weight pruning: make the model weights sparse– Structure pruning: remove the filter directly
• Knowledge Distillation– Use teacher model to guidesmaller student model– Student model is used for inferencing
• Quantization– Use less precision for model weights
• Adversarial attacks and brittleness?
NeurIPS/VIPER December 13, 2019 13
Applications That Need Low Power Deep Learning
• Health care monitoring
• Biomedical image analysis
• Image Based Phenotyping
NeurIPS/VIPER December 13, 2019 14
www.tadaproject.org
NeurIPS/VIPER December 13, 2019 15
Health Care Monitoring
• In 2015, the world spent $7.7 trillion on healthcare
• 6 out of 10 leading causes of death in US are related to diet (e.g., cancer, diabetes)
• Understanding the dietary patterns behind these causes is of great importance
NeurIPS/VIPER December 13, 2019 16
Technology Assisted Dietary Assessment (TADA)
• Traditional methods of tracking diet are inaccurate and labor-intensive
– Consists of self-reporting and record keeping
• In recent years, researchers have leveraged mobile phones and the Internet to collect images of food
• These images are analyzed to extract nutritional information to monitor a person’s diet
NeurIPS/VIPER December 13, 2019 17
Meal Image
Locate &Identify
Peach Ketchup Coke Milk
Hamburger French Fries Sugar Cookie
NeurIPS/VIPER December 13, 2019 18
User Studies
• We have completed a total of more than 50 user studies– Free-living environment– More than 5000 participants– More than 400,000 images acquired
• Each food image captures a real eating scene consists ofmultiple food items
NeurIPS/VIPER December 13, 2019 19
mFRCloud
Image(s) + Metadata (Geolocation, Time, Barcode, Contextual Info)
VolumeEstimation
- Labeled Images With Food Type (e.g. Milk, Toast, Eggs)
User Feedback (Confirmation or Correction)
Researchers,Practitioners
1 2 3
4
5
6
7
- Food Label Type- Segmented Image- Machine Learning- Context Processing - User Eating Patterns
Output
Wi-Fi/3G/4G/5G Network
Internet
TADA System Overview
Wi-Fi/3G/4G/5G Network
www.tadaproject.org
NeurIPS/VIPER December 13, 2019 20
TADA Image Analysis System
Image +Metadata
Food LocalizationFaster RCNN
Localized Food
Regions
Trained CNN
Food Labels
Food Segmentation using Weakly Supervised TechniquesFuse
Information
Segmentation Map
Portion SizeEstimation
Energy and Nutrients
NeurIPS/VIPER December 13, 2019 21
TADA Image AnalysisDeep Learning Approach
FoodLocalization
Food Classification
FoodSegmentation
milk
pancakesausage
Original Image
milk
sausage pancake
NeurIPS/VIPER December 13, 2019 22
Need for Low Power Computing
• Our system is built on a cloud-based approach
• The relevant processing happens on a remote machine and real-time feedback to the user is difficult
• If a similarly performing system could be operated on commonly used mobile phones, the user could take more direct control of their diet and how it interacts with other factors
NeurIPS/VIPER December 13, 2019 23
Image Based Phenotyping
NeurIPS/VIPER December 13, 2019 24
Definitions
• Phenotype:“any measurable characteristic or trait of a plant, and a result of combination of genes, environmental influence, and their interactions” or
“quantitative description of the plant’s anatomical, ontogenetical, physiological and biochemical properties”
• Phenotyping:“characterizing the performance of the plants for desired trait(s)”
NeurIPS/VIPER December 13, 2019 25
Traits• Some traits measured when phenotyping:
– Color– Height– Architecture (shape)– Canopy temperature– Canopy aperture– Water/nitrogen use efficiency– Number of leafs– Total leaf area– Grain yield– Fluorescence intensity
NeurIPS/VIPER December 13, 2019 26
Traditional and Modern Phenotyping
Traditional plant phenotyping
Automatedphenotyping in the field
UAV imagerydata
NeurIPS/VIPER December 13, 2019 27
Sensor Platforms
• Images are acquired from drones and the Phenorover
RGB LiDAR APX-15
NeurIPS/VIPER December 13, 2019 28
Orthophoto Generation Plot Extraction
Parameter files
Original imagesOrthophoto Plot Manual Refinement
Inverse Mapping Row Selection Destructive Sampling Mask Creation
Analysis per row Analysis per plant
Orthophoto visualization
Data Collection
Leaf Count
Canopy Coverage
Plant Centers
Plant Count
Largest Leaf’sLength and Width In a Row
Ground Reference Data Comparison
APSIM Model Fitting
Plant Center Ground Truth Labeling
Panel Corner Coordinates
Segment Coordinates
Segment Coordinates
Segment images inoriginal resolution
Leaf Segmentation
Leaf Count
Largest Leaf’sLength and Width and Area
Leaf Segmentation
Intra-row Distance
Largest Leaf’sLength and Width In a Row
Segment images to process Mask for each Segment
Traits Visualization
Plant Center Ground Truth
Plant Center Ground Truth
RGB Image Analysis
Prof. Tuinstra’s groupProf. Delp’s group
Prof. Habib’s group
Prof. Crawford’s group
Prof. Ebert’s group
Phenosorg RGB Data Processing Flow Chart
Traits for everything in
XML
Convert Selected ResultsTo Desired Format
Selected Rows
NeurIPS/VIPER December 13, 2019 29
Image Analysis System
Plot Extraction
Inverse Mapping
Traits Analysis(e. g., plant count,
plant location, leaf count)
Visualizationof Traits
Orthophoto Generation
TraitsPer Plant or Plot
Subrows(Original Resolution)
Subrows(Coordinates Of Vertices)
Orthophoto
NeurIPS/VIPER December 13, 2019 30
Orthophoto
Example of a fully rectified image of the entire field(orthomosaic)
07/15/2015
NeurIPS/VIPER December 13, 2019 31
Counting PlantsWith Deep Learning
NeurIPS/VIPER December 13, 2019 32
Dataset1,240 cigars ⇒ 1,240 images
• We extract one-row plots with our plot extraction tool
06/21/2016
NeurIPS/VIPER December 13, 2019 33
Plant Location UsingFully Convolutional Networks
• A Fully Convolutional Network (FCN) isany network that only contains convolutional layers
• This means the output of the FCN is another image
• The architecture of a FCN can be designed such that the output is of the same size as the input
• This is used in U-Net, a popular FCN architecture,to perform pixelwise segmentation
FCN Image
NeurIPS/VIPER December 13, 2019 34
FCN Architecture
• It accepts images of any size (bigger than 256x256)
𝒑𝒑𝒙𝒙(Probability Map)
�𝒀𝒀(Plant Count
Estimate)
Gaussian Mixture Model
�𝒀𝒀
NeurIPS/VIPER December 13, 2019 35
Cost Function For Plant Location
• The 𝒍𝒍𝟏𝟏 norm does not work well for localization tasks
• Let 𝑿𝑿 = {𝒙𝒙𝟏𝟏, … ,𝒙𝒙|�𝒀𝒀|} be the set of ground truth points, andlet 𝒀𝒀 = {𝒚𝒚𝟏𝟏, … ,𝒚𝒚|𝒀𝒀|} be the set of estimated locations
• A metric that measures the similarity between 𝑿𝑿 and 𝒀𝒀 isthe Hausdorff Distance:
𝒅𝒅𝑯𝑯 𝑿𝑿,𝒀𝒀 = max{max𝑥𝑥∈𝑋𝑋
min𝑦𝑦∈𝑌𝑌
𝑑𝑑(𝑥𝑥,𝑦𝑦) , max𝑦𝑦∈𝑌𝑌
min𝑥𝑥∈𝑋𝑋
𝑑𝑑(𝑥𝑥,𝑦𝑦)}
𝑑𝑑𝐻𝐻 𝑋𝑋,𝑌𝑌 = 𝑑𝑑(𝑥𝑥4,𝑦𝑦2)𝑥𝑥1
𝑦𝑦1𝑦𝑦2
𝑦𝑦3𝑦𝑦4
𝑥𝑥2 𝑥𝑥3
𝑥𝑥4
𝑥𝑥5
NeurIPS/VIPER December 13, 2019 36
Plant Location Cost Function
• In fact, our cost metric is a little more complicatedto deemphasize outliers and make it differentiablewith respect to the output of the FCN, 𝒑𝒑 ∈ 0, 1 :
𝑑𝑑𝑊𝑊𝐻𝐻 𝑝𝑝,𝑌𝑌 = 1∑𝑥𝑥∈Ω 𝑝𝑝𝑥𝑥
∑𝑥𝑥∈Ω 𝑝𝑝𝑥𝑥 min𝑦𝑦∈𝑌𝑌
𝑑𝑑(𝑥𝑥,𝑦𝑦) +
1𝑌𝑌∑𝑥𝑥∈Ωmin
𝑥𝑥∈Ω𝑑𝑑(𝑥𝑥,𝑦𝑦)𝑝𝑝𝑥𝑥𝛼𝛼
• The parameter 𝜶𝜶 balances precision and recall• This is similar to the Average Hausdorff Distance
𝑑𝑑𝐴𝐴𝐻𝐻 𝑋𝑋,𝑌𝑌 = 1𝑋𝑋∑𝑥𝑥∈𝑋𝑋 min
𝑦𝑦∈𝑌𝑌𝑑𝑑(𝑥𝑥,𝑦𝑦) + 1
𝑌𝑌∑𝑦𝑦∈𝑌𝑌 min
𝑥𝑥∈𝑋𝑋𝑑𝑑(𝑥𝑥,𝑦𝑦)
NeurIPS/VIPER December 13, 2019 37
Results (Metrics)• We report the following metrics in 𝟐𝟐𝟐𝟐𝟐𝟐 × 𝟐𝟐𝟐𝟐𝟐𝟐 images
– Accuracy (% of estimations at ≤ 𝒓𝒓 pixels to a plant)– Average Hausdorff Distance– Mean Average Percent Error
CCx −
=ˆ
100MAPE
MetricAHD 8.8 px
MAPE 4 %
Acc
urac
y (%
)
𝒓𝒓 (pixels)𝟐𝟐
𝟗𝟗0%
NeurIPS/VIPER December 13, 2019 38
Regression Term
• We modified the regression term in our cost function
𝑑𝑑𝑊𝑊𝐻𝐻𝑊𝑊 𝑝𝑝,𝑌𝑌 =1
∑𝑥𝑥∈Ω 𝑝𝑝𝑥𝑥�𝑥𝑥∈Ω
𝑝𝑝𝑥𝑥 min𝑦𝑦∈𝑌𝑌
𝑑𝑑(𝑥𝑥, 𝑦𝑦) +1𝑌𝑌
�𝑥𝑥∈Ω
min𝑥𝑥∈Ω
𝑑𝑑(𝑥𝑥,𝑦𝑦)𝑝𝑝𝑥𝑥𝛼𝛼
+ 𝐿𝐿𝑟𝑟𝑟𝑟𝑟𝑟( �𝑌𝑌 − 𝑌𝑌 )
– We use Huber loss (also known as smooth L1)
– Advantages:Differentiable at the origin, anddoes not over-penalize outliers
• We achieved an F-score: 93 % (at 5cm of error)
NeurIPS/VIPER December 13, 2019 39
Results: 2016 Data
Estimated plant centers (in red)
NeurIPS/VIPER December 13, 2019 40
Results: 2017 Data
• Trained our model using ground truthof the Inbred Calibration Panel by Neal Carpenter
• Mean Average Percent Error: 12%• Mean Average Error:
2.6 plants per row• F-score: 93 %
(at 5cm error)
Testing results on the Hybrid Calibration Panel (2017/06/09)
NeurIPS/VIPER December 13, 2019 41
Need for Low Power Computing
• End-users may not have the computational resource for fast inferencing
• Need for re-training
• Farmers need portable devices to get the results
• Edge-based computing is needed in this situation
NeurIPS/VIPER December 13, 2019 42
NeurIPS/VIPER December 13, 2019 43
Acknowledgements
NeurIPS/VIPER December 13, 2019 44
Cheap, Fast, and Low Power Deep Learning: I need it now!(Please and Thank You)
Edward J. Delp
Video and Image Processing Laboratory (VIPER)School of Electrical and Computer Engineering
School of Biomedical Engineering Purdue University
West Lafayette, Indiana
email: ace@ecn.purdue.eduhttps://engineering.purdue.edu/~ace/
top related