structured predictions with deep learninghays/compvision2017/lectures/35.pdf · convolutional pose...

Structured Predictions with Deep Learning James Hays

Upload: others

Post on 17-Jul-2020

1 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Structured Predictions with Deep Learning

James Hays

Recap of previous lecture

• COCO dataset. Instance segmentation of 80 categories. Keypoints + Language + other annotations, as well.

• Deeper deep models• VGG networks

• GoogLeNet built from “Inception” modules

• ResNet

• Deeper networks seem to work better than the equivalent shallow network with the same number of parameters, but they aren’t trivial to train.

Page 3: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

“tabby cat”

a classification network

Fully Convolutional Networks for Semantic Segmentation.

Jon Long, Evan Shelhamer, Trevor Darrell. CVPR 2015

Page 4: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

becoming fully convolutional

Note: “Fully Convolutional” and “Fully Connected” aren’t the same thing.

They’re almost opposites, in fact.

Page 5: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

becoming fully convolutional

Page 6: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

upsampling output

Page 7: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

end-to-end, pixels-to-pixels network

Page 8: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

What if we want other types of outputs?

• Easy: Predict any number of labels (with classification, there will be just one best answer, but for other labels like attributes dozens could be appropriate for an image)

Page 9: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

What if we want other types of outputs?

• Easy*: Predict any fixed dimensional output, whether a feature (embedding networks) or an image.

Scribbler: Controlling Deep Image Synthesis with Sketch and Color. Sangkloy, Lu, Chen Yu, and Hays. CVPR 2017

*easy to design an architecture. Not necessarily easy to get working.

Page 10: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

What if we want other types of outputs?• Hard: Outputs with varying dimensionality or

cardinality• A natural language image caption

• An arbitrary number of human keypoints (17 points each)

• An arbitrary number of bounding boxes (4 parameters each)

• Today we will examine state-of-the-art methods for keypoint prediction and object detection

Convolutional Pose Machines

• Variant of Convolutional Pose Machines that won the inaugural COCO keypoint challenge.

• http://image-net.org/challenges/talks/2016/Multi-person%20pose%20estimation-CMU.pdf

• Videos: https://www.youtube.com/playlist?list=PLNh5A7HtLRcpsMfvyG0DED-Dr4zW5Lpcg

Multi-Person Pose Estimation using

Part Affinity FieldsZhe Cao, Shih-En Wei, Tomas Simon, Yaser Sheikh

Carnegie Mellon University

http://image-net.org/challenges/talks/2016/Multi-person pose estimation-CMU.pdf

https://www.youtube.com/playlist?list=PLNh5A7HtLRcpsMfvyG0DED-Dr4zW5Lpcg

Page 12: Structured Predictions with Deep Learninghays/compvision2017/lectures/35.pdf · Convolutional Pose Machines •Variant of Convolutional Pose Machines that won the inaugural COCO keypoint

SSDBox

• Object Detector that is very nearly state-of-the-art accuracy and very, very fast

• http://www.cs.unc.edu/~wliu/papers/ssd_eccv2016_slide.pdf

http://www.cs.unc.edu/~wliu/papers/ssd_eccv2016_slide.pdf

Convolutional Pose Machines - cv-foundation.org · Convolutional Pose Machines Shih-En Wei ... we show a systematic design for how convolutional net- ... The pose machine [29] is

Pose2Mesh: Graph Convolutional Network for 3D Human Pose

Articulate Object Keypoint Detection and Pose Estimation · Convolutional Pose Machines provide a networksequence for leaning rich implicit models.It’s a deep learning architecturelearn

ConvPoseCNN: Dense Convolutional 6D Object Pose EstimationConvPoseCNN: Dense Convolutional 6D Object Pose Estimation Catherine Capellen 1, Max Schwarz a and Sven Behnke b 1Autonomous

Deep ChArUco: Dark ChArUco Marker Pose Estimation · 2019. 6. 10. · marker’s 6DoF pose. ChArUcoNet is a two-headed marker-speciﬁc convolutional neural network (CNN) which jointly

Convolutional Pose Machines: A Deep Architecture for ... · We introduce Convolutional Pose Machines (CPMs) for the task of articulated pose estimation. CPMs inherit the beneﬁts

Leveraging Convolutional Pose Machines for Fast and ...odobez/publications/CaoCanevetOdobez-IROS2018.pdf · II. CONVOLUTIONAL POSE MACHINES The ﬁrst part of our head pose estimation

Convolutional Pose Machines - arXiv · Convolutional pose machines consist of a sequence of prediction networks trained to predict conﬁdences for the locations of each part, improving

Pose Estimation - Virginia Tech · Convolutional Pose Machines 1. Capture local appearance with CNNs 3. Iteratively refine confidence maps with global cues 11 2. CNNs on confidence

Learning Feature Pyramids for Human Pose Estimationis.ulsan.ac.kr/files/announcement/627/Learning Feature Pyramids for... · tion, convolutional pose machines [55] and stacked hour-glass

Deep Convolutional Neural Networks for E cient Pose …vgg/publications/2014/... · 2014-11-02 · Deep Convolutional Neural Networks for ... As the regressor we use a convolutional

An Efﬁcient Convolutional Network for Human … KOSTRIKOV, GALL, LEIBE: EFFICIENT CNN FOR HUMAN POSE ESTIMATION 1 An Efﬁcient Convolutional Network for Human Pose Estimation Umer

Learning Human Pose Estimation Features with Convolutional ... · to build better detectors. 3 Model To perform pose estimation with a convolutional network architecture [24] (convnet),

Towards 3D Human Pose Estimation in the Wild: a …‘¨星壹.pdfWei et al. Convolutional Pose Machines Newell et al. Hourglass Network Bulat et al. Part Heatmap Regression 2D pose

Recurrent Human Pose Estimationvgg/publications/2017/Belagiannis17/belagian… · In addition, our model shares with Convolutional Pose Machines [43] and the Hourglass model [25]

An Efﬁcient Convolutional Network for Human Pose Estimation · In recent years, human pose estimation has greatly beneﬁted from deep learning and huge gains in performance have

Real-Time Head Pose Estimation with Convolutional Neural ...cs231n.stanford.edu/reports/2015/pdfs/zhianghu_report.pdf · Real-Time Head Pose Estimation with Convolutional Neural Networks

Weakly-Supervised 3D Pose Estimation from a …epubs.surrey.ac.uk/852639/1/Weakly-Supervised 3D Pose...More recently, Convolutional Pose Machines have become a popular approach [34]

Urban Localization with Street Views Using a Convolutional ......Urban Localization with Street Views using a Convolutional Neural Network for End-to-End Camera Pose Regression Guillaume

Total Capture: 3D Human Pose Estimation Fusing Video and … · 2017. 8. 16. · TOTAL CAPTURE: POSE ESTIMATION FUSING VIDEO AND IMU DATA 3. simple 2D convolutional neural network

Weakly-supervised 3D Hand Pose Estimation from Monocular … · 2019-04-30 · Convolutional Pose Machines [Wei. et al. CVPR 2016] Stacked Hourglass Networks [Newell et al. ECCV 2016]

Real-Time Continuous Pose Recovery of Human Hands ...tompson/others/TOG_2014_paper...PREPRINT Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks !

Hand Detection and Pose Estimation using Convolutional ...859561/FULLTEXT01.pdf · Convolutional networks was ﬁrst applied with great success to image classiﬁ- cation, sparking

3D Pose Regression using Convolutional Neural Networks · 2017. 8. 21. · 3D Pose Regression using Convolutional Neural Networks Siddharth Mahendran [email protected] Haider Ali

Real-time Convolutional Networks for Depth-based Human Pose …odobez/publications/MartinezVillamizarCane... · 2018-08-16 · Real-time Convolutional Networks for Depth-based Human

Joint Training of a Convolutional Network and a …...Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation Jonathan Tompson, Arjun Jain, Yann LeCun,

Deep Convolutional Neural Networks for E cient Pose Estimation in Gesture …vgg/publications/2014/Pfister14a/... · 2014. 11. 2. · Deep ConvNets for E cient Pose Estimation in

Recognizing Human Actions as the Evolution of Pose ... · Figure 2: The overview of the proposed method. a) Convolutional pose machines predict pose estimation map of each body part

DeepPose & Convolutional Pose Machinescseweb.ucsd.edu/.../Presentations/12_Human_Pose_Estimation_Rant… · DeepPose & Convolutional Pose Machines. Main Concepts 1. CNN with regressor

Weakly-supervised 3D Hand Pose Estimation from Monocular ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/31_Jan… · Convolutional Pose Machines [Wei. et al. CVPR 2016]

Convolutional Networks for Object Category and 3D Pose ... · Convolutional Networks for Object Category and 3D Pose Estimationfrom 2D Images Siddharth Mahendran, Haider Ali, and

Convolutional Pose Machines: A Deep Architecture …We introduce Convolutional Pose Machines (CPMs) for the task of articulated pose estimation. CPMs inherit the beneﬁts of the pose

Relative Camera Pose Estimation Using … Camera Pose Estimation Using Convolutional Neural Networks Iaroslav Melekhov 1, Juha Ylioinas , Juho Kannala , and Esa Rahtu2 1 Aalto University,

Machine Learning for Video Tracking of Passengers and ...Convolutional Pose Machine [1] Mask R-CNN [2] [1] Shih-En, Ramakrishna, Kanade, Sheikh. "Convolutional pose machines." IEEE

3D Convolutional Neural Networks for Efficient and Robust Hand … · 2017. 5. 31. · 3D Convolutional Neural Networks for Efﬁcient and Robust Hand Pose Estimation from Single