faster r-cnn: towards real-time object detection with region proposal networks (upc reading group)
TRANSCRIPT
![Page 1: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/1.jpg)
Faster R-CNN:Towards Real-Time Object Detection with
Region Proposal Networks
Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun[paper@NIPS15][arXiv][python][matlab][slides by R. Girshick]
Slides by Amaia Salvador [GDoc]Computer Vision Reading Group (01/03/2016)
![Page 2: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/2.jpg)
1. Introduction
![Page 3: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/3.jpg)
Object Detection
3
![Page 4: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/4.jpg)
Object Detection: Previously...
DPM. P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan. Object Detection with Discriminatively Trained Part Based Models. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, Sep. 2010
DPM
4
Hand-crafted features + Sliding Window
![Page 5: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/5.jpg)
Object Detection: Previously...
R-CNN. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014, June). Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on(pp. 580-587). IEEE.SPPnet. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Spatial pyramid pooling in deep convolutional networks for visual recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 37(9), 1904-1916.Fast R-CNN. Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1440-1448).
R-CNN SPPnet Fast R-CNN
5
CNN features + Object Proposals
![Page 6: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/6.jpg)
Object Detection: Limitations
Selective Search CPMC
MCG
Object Proposal computation is the bottleneck in current state of the art object detection systems
Selective Search. Van de Sande, K. E., Uijlings, J. R., Gevers, T., & Smeulders, A. W. (2011, November). Segmentation as selective search for object recognition. InComputer Vision (ICCV), 2011 IEEE International Conference on (pp. 1879-1886). IEEE.CPMC. Carreira, J., & Sminchisescu, C. (2010, June). Constrained parametric min-cuts for automatic object segmentation. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on (pp. 3241-3248). IEEE.MCG. Arbeláez, P., Pont-Tuset, J., Barron, J., Marques, F., & Malik, J. (2014). Multiscale combinatorial grouping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 328-335). 6
![Page 7: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/7.jpg)
Faster R-CNN: Motivation
Selective Search CPMC
MCG
Replace the usage of external Object Proposals with a Region Proposal Network (RPN).
7
![Page 8: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/8.jpg)
2. Methodology
![Page 9: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/9.jpg)
Faster R-CNN: Overview
Conv Layer 5
Co
nv
laye
rsRPN RPN Proposals
RPN Proposals
Class probabilities
RoI pooling layerFC layersClass scores
9
![Page 10: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/10.jpg)
Faster R-CNN: Overview
Conv Layer 5
Co
nv
laye
rsRPN RPN Proposals
RPN Proposals
Class probabilities
RoI pooling layerFC layersClass scores
10
![Page 11: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/11.jpg)
Region Proposal Network (RPN)
11
Objectness scores
Bounding Box Regression
In practice, k = 9 (3 different scales and 3 aspect ratios)
![Page 12: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/12.jpg)
RPN: Loss Function
12
Predicted probability of being an object for anchor i
i = anchor index in minibatch
Coordinates of the predicted bounding box for anchor i
Ground truth objectness label
True box coordinates
Ncls
= Number of anchors in minibatch (~ 256)Nreg = Number of anchor locations ( ~ 2400)
Log lossSmooth L1 loss
In practice = 10, so that both terms are roughly equally balanced
![Page 13: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/13.jpg)
RPN: Positive/Negative Samples
13
An anchor is labeled as positive if:
(a) the anchor is the one with highest IoU overlap with a ground-truth box
(b) the anchor has an IoU overlap with a ground-truth box higher than 0.7
Negative labels are assigned to anchors with IoU lower than 0.3 for all ground-truth boxes.
50%/50% ratio of positive/negative anchors in a minibatch.
![Page 14: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/14.jpg)
Faster R-CNN: Overview
Conv Layer 5
Co
nv
laye
rsRPN RPN Proposals
RPN Proposals
Class probabilities
RoI pooling layerFC layersClass scores
14
![Page 15: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/15.jpg)
Object Detection Network
15Fast R-CNN
![Page 16: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/16.jpg)
Object Detection Network: Loss
16
*From Fast R-CNN
Predicted class scores
True class scores
True box coordinates
Predicted box coordinates
Log lossSmooth L1 loss
![Page 17: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/17.jpg)
Fast R-CNN: Positive/Negative Samples
17
Positive samples are defined as those whose IoU overlap with a ground-truth bounding box is > 0.5.
Negative examples are sampled from those that have a maximum IoU overlap with ground truth in the interval [0.1, 0.5).
25%/75% ratio for positive/negative samples in a minibatch.
*From Fast R-CNN
![Page 18: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/18.jpg)
Faster R-CNN: Training
18
Conv Layer 5
Co
nv
laye
rsRPN RPN Proposals
RPN Proposals
Class probabilities
RoI pooling layerFC layersClass scores
4-step training to share features for RPN and Fast R-CNN
![Page 19: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/19.jpg)
Faster R-CNN: 4-step training
Conv Layer 5
Co
nv
laye
rsRPN RPN Proposals
19
Step 1: Train RPN initialized with an ImageNet pre-trained model.
ImageNet weights(fine tuned)
![Page 20: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/20.jpg)
Faster R-CNN: 4-step training
Conv Layer 5
Co
nv
laye
rs
RPN Proposals (learned in 1)
Class probabilities
20
Step 2: Train Fast R-CNN with learned RPN proposals.
ImageNet weights(fine tuned)
![Page 21: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/21.jpg)
Faster R-CNN: 4-step training
Conv Layer 5
Co
nv
laye
rsRPN RPN Proposals
21
Step 3: The model trained in 2 is used to initialize RPN and train again.
Weights from Step 2(fixed)
![Page 22: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/22.jpg)
Faster R-CNN: 4-step training
Conv Layer 5
Co
nv
laye
rs
RPN Proposals (learned in 3)
Class probabilities
22
Step 4: Fine tune FC layers of Fast R-CNN using same shared convolutional layers as in 3.
Weights from Step 2&3(fixed)
![Page 23: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/23.jpg)
3. Experiments
![Page 24: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/24.jpg)
Experiments: CNN Architectures
24
VGG-16: Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
ZF: Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In Computer vision–ECCV 2014 (pp. 818-833). Springer International Publishing.
![Page 25: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/25.jpg)
Experiments: Datasets
25
![Page 26: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/26.jpg)
Experiments I: VOC 2007 & ZF
26
Comparison between Fast R-CNN trained with external object proposals (SS: Selective Search, EB: EdgeBoxes) with Faster R-CNN
![Page 27: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/27.jpg)
Experiments I: VOC 2007 & ZF
27
![Page 28: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/28.jpg)
Experiments I: VOC 2007 & ZF
28
![Page 29: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/29.jpg)
Experiments I: VOC 2007 & ZF
29
![Page 30: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/30.jpg)
Experiments II
30
Detection Accuracy
Timing (ms)
![Page 31: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/31.jpg)
Experiments III
31
![Page 32: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/32.jpg)
Experiments IV
32
One-Stage Detection: 1) Directly Refine and Classify Sliding Window locations
Two-Stage Proposal + Detection: 1) Learn Object Proposals 2) Refine and classify Object Proposals
![Page 33: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/33.jpg)
Experiments V: MS COCO (arXiv)
33
![Page 34: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/34.jpg)
Qualitative Results
34
![Page 35: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/35.jpg)
4. Summary
![Page 36: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/36.jpg)
Summary
36
● Region Proposal Network sharing convolutional features with Object Detection Network makes region generation step nearly cost-free.
● Quality of proposals is improved with RPN wrt SS and EB.
● Object Detection system at 5-17 fps.
![Page 37: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/37.jpg)
Summary
37
● Faster R-CNN is the basis of the winners of COCO and ILSVRC 2015 object detection competitions [1].
● RPN is also used in the winning entries of ILSVRC 2015 localization [1] and COCO 2015 segmentation competitions [2].
[1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” arXiv:1512.03385, 2015.[2] J. Dai, K. He, and J. Sun, “Instance-aware semantic segmentation via multi-task network cascades,” arXiv:1512.04412, 2015.
![Page 38: Faster R-CNN: Towards real-time object detection with region proposal networks (UPC Reading Group)](https://reader031.vdocument.in/reader031/viewer/2022030317/5870d7611a28ab64768b6e15/html5/thumbnails/38.jpg)
Thank you !
Questions?