final presentation andy rosales (1) (1)
TRANSCRIPT
![Page 1: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/1.jpg)
Where is the bear?
Andy Rosales EliasMentor: Nevena Golubovic
Advisors: Chandra Krintz, Rich WolskiLab: RACELab
Department of Computer Science
Implementing Machine Learning-Based Image Recognition for Animal Detection
![Page 2: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/2.jpg)
Carters New Agency http://www.catersnews.com/
![Page 3: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/3.jpg)
Carters New Agency http://www.catersnews.com/
![Page 4: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/4.jpg)
The Problem of Accuracy, Cost, And Efficiency
= + +
![Page 5: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/5.jpg)
Data Reliability
The Problem of Accuracy, Cost, And Efficiency
= + +
![Page 6: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/6.jpg)
Animal Recognition Using Different Machine Learning Frameworks
Output: “Bear” Output: “None”
![Page 7: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/7.jpg)
Using Convolutional Neural Networks (CNN) to Recognize Animals
-Zhong, Zhuoyao et al. “High Performance Offline Handwritten Chinese Character Recognition Using GoogLeNet and Directional Feature Maps.” -"Multi-column deep neural networks for image classification". 2012 IEEE Conference on Computer Vision and Pattern -Recognition-Ciresan, Dan; Ueli Meier; Jonathan Masci; Luca M. Gambardella; Jurgen Schmidhuber. "Flexible, High Performance Convolutional Neural Networks for Image Classification"
- Almost every highly ranked team used CNN as their basic framework
- Foundation of GoogLeNet and AlexNet
![Page 8: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/8.jpg)
Dog
Cat
How Convolutional Neural Networks Classify Images
Feature extractor black box
Deactivated neuron
Activated neuron
}
}
Input layer
Characteristic layers
Output layer
![Page 9: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/9.jpg)
How Convolutional Neural Networks Classify Images
Feature extractor black box
Dog
Cat
Deactivated neuron
Activated neuron
}
}
Input layer
Characteristic layers
Output layer
![Page 10: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/10.jpg)
How Convolutional Neural Networks Classify Images
Feature extractor black box
Dog
Cat
Deactivated neuron
Activated neuron
}
}
Input layer
Characteristic layers
Output layer
![Page 11: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/11.jpg)
How Convolutional Neural Networks Classify Images
Feature extractor black box
Dog
Cat
Deactivated neuron
Activated neuron
}
}
Input layer
Characteristic layers
Output layer
![Page 12: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/12.jpg)
Re-training an Existing CNN Model
Deactivated neuron
Activated neuron
● Use a pre-trained CNN model
Tree
Deer
Pond
Characteristic layers(pre-trained model)
![Page 13: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/13.jpg)
Re-training an existing CNN Model
Deactivated neuron
Activated neuron
“deer”
Extra filtering layer
● Use a pre-trained CNN model● Add an extra filtering layer
Characteristic layers(pre-trained model)
![Page 14: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/14.jpg)
Results After Classifying 12210 Images Using Pre-trained CNN Model
Gazelle 9%
Mountain bike - 10%
TensorFlow - Top 5 Categories
Barrow17%
MountainTent38%
Other - 26%
![Page 15: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/15.jpg)
Results After Classifying 12210 Images Using TensorFlow Re-trained In Four Categories
Empty/other - 76%
Deer - 19%Coyote 3%
Bear0.5%
Re-training Dataset:● Bear: 170 images● Deer: 489 images● Coyote: 259 images● Empty/other: 21 images
Re-trained CNN: Tensorflow
![Page 16: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/16.jpg)
Accuracy of TensorFlow: Before And After Re-Training
No Re-training Retrained in four categories
Average accuracy: 35% Average accuracy: 73%
![Page 17: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/17.jpg)
Metadata Extraction & Fusion
● Learn more about how animals are affected by the environment.
● Analyze this data to learn more about certain species.
![Page 18: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/18.jpg)
● Time and date can be extracted from the EXIF data● Temperature needs to be recognized using OCR
Extracting metadata from images
![Page 19: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/19.jpg)
Recognizing temperature digits using OCRHave temperature digits recognized automatically
![Page 20: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/20.jpg)
Recognizing temperature digits using OCRHave temperature digits recognized automatically
? ?
![Page 21: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/21.jpg)
Training phase - labeling digitsLabel each digit in the font family in order to train the model
![Page 22: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/22.jpg)
Training phase - labeling digits
0
Manual labeling
Label each digit in the font family in order to train the model
![Page 23: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/23.jpg)
Training phase - labeling digits
0 1
Manual labeling
Label each digit in the font family in order to train the model
![Page 24: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/24.jpg)
Training phase - labeling digits
0 1 2
Manual labeling
Label each digit in the font family in order to train the model
![Page 25: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/25.jpg)
Label each digit in the font family in order to train the model
Training phase - labeling digits
0 1 2 3 4 5 6 7 8 9 -
Manual labeling
![Page 26: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/26.jpg)
Recognizing temperature digits using OCR
● The trained model will recognize any unlabeled character● Which number is the “nearest” in similarity with the labeled data
9 4
![Page 27: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/27.jpg)
Collecting metadata with hidden, motion-triggered cameras at the Sedgwick Reserve
Camera 1 Camera 2 Camera 3
![Page 28: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/28.jpg)
Camera 1 Camera 2 Camera 3
Collecting metadata with hidden, motion-triggered cameras at the Sedgwick Reserve
![Page 29: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/29.jpg)
UnknownClassifier
84
1319
How each OCR algorithm processes temperature images
Previous OCR
![Page 30: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/30.jpg)
UnknownClassifier
84
1319
Camera 1
Camera 2
Camera 3
94
73
79
Previous OCR
New OCR
How each OCR algorithm processes temperature images
![Page 31: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/31.jpg)
Analyzing the accuracy and processing time of both OCR algorithms
Accuracy % Time in seconds
![Page 32: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/32.jpg)
- Re-train TensorFlow with labeled images from Sedgwick Reserve- Collect more data to build a bigger dataset- Move everything to the cloud
The Road Ahead
Manual Labeling
Improve CNN Model
![Page 33: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/33.jpg)
AcknowledgmentsSpecial Thanks To:
Mentor: Nevena GolubovicAdvisors: Chandra Krintz and Rich WolskiProgram Coordinator: Stephanie Mendes
CSEP
![Page 34: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/34.jpg)
![Page 35: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/35.jpg)
![Page 36: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/36.jpg)
Recognizing temperature digits using OCR
● We need to automatically have these numbers recognized
![Page 37: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/37.jpg)
Recognizing temperature digits using OCR
● We need to automatically have these numbers recognized● Find the edges on the image
![Page 38: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/38.jpg)
Recognizing temperature digits using OCR
● We need to automatically have these numbers recognized● Find the edges on the image● With the edges, we know the dimensions of each digit/symbol
![Page 39: Final Presentation Andy Rosales (1) (1)](https://reader031.vdocument.in/reader031/viewer/2022022414/5877ec841a28ab20088b6359/html5/thumbnails/39.jpg)
Recognizing temperature digits using OCR
● We need to automatically have these numbers recognized● Find the edges on the image● With the dimensions, we ignore characters with small area (degree symbol)
● However, numbers are still unlabeled
? ?